2024-09-13T16:33:22.314 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T16:33:22.336 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T16:33:22.442 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903833 branch: wip-rishabh-testing-20240910.095512-reef description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} email: null first_in_suite: false job_id: '7903833' kernel: client: branch: testing kdb: 1 sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240910.095512-reef ceph: cephfs: max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: true mds_sleep_rank_change: 5000000.0 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 mon warn on pool no app: false osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - overall HEALTH_ - \(MON_DOWN\) sha1: 50bbe7f395997dcd871813949c49addbbf630361 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 50bbe7f395997dcd871813949c49addbbf630361 kclient: syntax: v2 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240910.095512-reef sha1: 50bbe7f395997dcd871813949c49addbbf630361 owner: scheduled_rishabh@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 1152 sha1: 50bbe7f395997dcd871813949c49addbbf630361 sleep_before_teardown: 0 subset: 984/1024 suite: fs suite_branch: wip-rishabh-testing-20240910.095512-reef suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_50bbe7f395997dcd871813949c49addbbf630361/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 50bbe7f395997dcd871813949c49addbbf630361 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLmqOJH6hrd6+FHScvw0KFTm9tf2H3Cjqo0IzxKbrZ+xzeRQM3ZbE2myVHjp7c6sn5WKtKzVVYwYtJIQWXgOufc= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPqJZWX7a+7f4/ul+wjpq2z8cqD/fdqYZyTtAprwnkX1mDrWyiVEz608p6BeRGsGSEnPccSeRTZWBlnnBPaDtmc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - mon_thrash: check_mds_failover: true revive_delay: 20 thrash_delay: 10 - workunit: clients: all: - suites/iozone.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_14:36:41 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T16:33:22.443 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_50bbe7f395997dcd871813949c49addbbf630361/qa; will attempt to use it 2024-09-13T16:33:22.443 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_50bbe7f395997dcd871813949c49addbbf630361/qa/tasks 2024-09-13T16:33:22.444 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T16:33:22.445 INFO:teuthology.task.internal:Checking packages... 2024-09-13T16:33:22.467 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '50bbe7f395997dcd871813949c49addbbf630361' 2024-09-13T16:33:22.467 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T16:33:22.468 INFO:teuthology.packaging:ref: None 2024-09-13T16:33:22.468 INFO:teuthology.packaging:tag: None 2024-09-13T16:33:22.468 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240910.095512-reef 2024-09-13T16:33:22.468 INFO:teuthology.packaging:sha1: 50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T16:33:22.468 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240910.095512-reef 2024-09-13T16:33:22.631 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-850.g50bbe7f3 2024-09-13T16:33:22.633 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T16:33:22.639 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T16:33:22.639 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T16:33:22.661 INFO:teuthology.task.internal:Saving configuration 2024-09-13T16:33:22.679 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T16:33:22.686 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T16:33:22.719 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903833', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 16:28:17.709492', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLmqOJH6hrd6+FHScvw0KFTm9tf2H3Cjqo0IzxKbrZ+xzeRQM3ZbE2myVHjp7c6sn5WKtKzVVYwYtJIQWXgOufc='} 2024-09-13T16:33:22.751 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903833', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 16:28:17.712049', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPqJZWX7a+7f4/ul+wjpq2z8cqD/fdqYZyTtAprwnkX1mDrWyiVEz608p6BeRGsGSEnPccSeRTZWBlnnBPaDtmc='} 2024-09-13T16:33:22.751 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T16:33:22.759 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-09-13T16:33:22.759 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-09-13T16:33:22.759 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T16:33:22.836 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1421e4b0a0>, signals=[15]) 2024-09-13T16:33:22.836 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T16:33:22.842 INFO:teuthology.task.internal:Opening connections... 2024-09-13T16:33:22.842 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T16:33:22.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:33:22.931 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2024-09-13T16:33:22.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:33:23.013 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T16:33:23.020 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-09-13T16:33:23.038 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-09-13T16:33:23.038 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-09-13T16:33:23.097 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-09-13T16:33:23.097 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-09-13T16:33:23.097 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-09-13T16:33:23.097 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-09-13T16:33:23.097 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-09-13T16:33:23.097 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-09-13T16:33:23.097 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T16:33:23.097 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-09-13T16:33:23.098 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-09-13T16:33:23.098 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T16:33:23.098 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-09-13T16:33:23.098 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T16:33:23.098 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T16:33:23.098 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T16:33:23.098 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-09-13T16:33:23.130 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2024-09-13T16:33:23.148 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2024-09-13T16:33:23.149 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2024-09-13T16:33:23.207 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2024-09-13T16:33:23.207 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9" 2024-09-13T16:33:23.207 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2024-09-13T16:33:23.207 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2024-09-13T16:33:23.207 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9" 2024-09-13T16:33:23.207 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9" 2024-09-13T16:33:23.207 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T16:33:23.207 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2024-09-13T16:33:23.207 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon" 2024-09-13T16:33:23.208 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T16:33:23.208 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2024-09-13T16:33:23.208 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T16:33:23.208 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T16:33:23.208 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T16:33:23.208 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2024-09-13T16:33:23.238 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T16:33:23.249 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T16:33:23.261 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T16:33:23.261 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-09-13T16:33:23.263 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2024-09-13T16:33:23.280 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T16:33:23.286 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T16:33:23.287 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-09-13T16:33:23.320 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2024-09-13T16:33:23.439 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T16:33:23.513 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T16:33:23.527 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'} 2024-09-13T16:33:23.527 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-09-13T16:33:23.527 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-09-13T16:33:23.527 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T16:33:23.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:33:23.545 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-09-13T16:33:23.728 DEBUG:teuthology.task.kernel:sha1 for client.0 is 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-13T16:33:23.728 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "2ca44a2f35ca8f36983aaaa46d49312239ef519a"... 2024-09-13T16:33:23.728 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-09-13T16:33:23.746 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-503.el9.x86_64 2024-09-13T16:33:23.746 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-503.el9.x86_64 vs 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-13T16:33:23.746 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-09-13T16:33:23.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum upgrade -y linux-firmware 2024-09-13T16:33:26.109 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-13T16:33:30.384 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 6.6 MB/s | 20 MB 00:03 2024-09-13T16:33:35.226 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 7.5 MB/s | 6.5 MB 00:00 2024-09-13T16:33:36.736 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 19 kB 00:00 2024-09-13T16:33:37.932 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-13T16:33:43.529 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-13T16:33:45.683 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-13T16:33:45.684 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-09-13T16:33:45.684 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-13T16:33:45.791 INFO:teuthology.task.kernel:Downloading kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.0... 2024-09-13T16:33:45.791 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-13T16:33:45.951 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ 2024-09-13T16:33:45.952 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ --input-file=- 2024-09-13T16:33:45.954 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi028.front.sepia.ceph.com to complete... 2024-09-13T16:33:52.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13 16:33:52 URL:https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm [359633503/359633503] -> "/tmp/kernel.x86_64.rpm" [1] 2024-09-13T16:33:52.671 INFO:teuthology.orchestra.run.smithi028.stderr:FINISHED --2024-09-13 16:33:52-- 2024-09-13T16:33:52.671 INFO:teuthology.orchestra.run.smithi028.stderr:Total wall clock time: 6.7s 2024-09-13T16:33:52.672 INFO:teuthology.orchestra.run.smithi028.stderr:Downloaded: 1 files, 343M in 6.4s (53.6 MB/s) 2024-09-13T16:33:52.674 INFO:teuthology.task.kernel:Installing kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.0... 2024-09-13T16:33:52.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-09-13T16:33:54.884 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying... ######################################## 2024-09-13T16:33:55.003 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing... ######################################## 2024-09-13T16:33:55.133 INFO:teuthology.orchestra.run.smithi028.stdout:Updating / installing... 2024-09-13T16:34:03.453 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-6.11.0_rc5_g2ca44a2f35ca-1 ######################################## 2024-09-13T16:34:43.102 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-09-13T16:34:43.103 DEBUG:teuthology.orchestra.run.smithi028:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-09-13T16:34:43.201 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/System.map-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:34:43.202 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/config-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:34:43.202 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/vmlinuz-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:34:43.202 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:34:43.202 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/System.map 2024-09-13T16:34:43.202 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/config 2024-09-13T16:34:43.202 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel 2024-09-13T16:34:43.202 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch 2024-09-13T16:34:43.202 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86 2024-09-13T16:34:43.202 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto 2024-09-13T16:34:43.203 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-09-13T16:34:43.203 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-09-13T16:34:43.203 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-09-13T16:34:43.203 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-09-13T16:34:43.203 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-09-13T16:34:43.203 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-09-13T16:34:43.203 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-09-13T16:34:43.203 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32c-intel.ko 2024-09-13T16:34:43.203 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-09-13T16:34:43.203 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-09-13T16:34:43.203 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-09-13T16:34:43.204 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-09-13T16:34:43.204 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx2.ko 2024-09-13T16:34:43.204 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-09-13T16:34:43.204 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-09-13T16:34:43.204 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-09-13T16:34:43.204 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-09-13T16:34:43.204 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-09-13T16:34:43.204 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-09-13T16:34:43.204 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-09-13T16:34:43.204 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel 2024-09-13T16:34:43.204 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu 2024-09-13T16:34:43.205 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce 2024-09-13T16:34:43.205 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-09-13T16:34:43.205 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm 2024-09-13T16:34:43.205 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-amd.ko 2024-09-13T16:34:43.205 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-intel.ko 2024-09-13T16:34:43.205 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm.ko 2024-09-13T16:34:43.205 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto 2024-09-13T16:34:43.205 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ansi_cprng.ko 2024-09-13T16:34:43.205 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/anubis.ko 2024-09-13T16:34:43.205 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/arc4.ko 2024-09-13T16:34:43.205 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx 2024-09-13T16:34:43.205 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_memcpy.ko 2024-09-13T16:34:43.205 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_pq.ko 2024-09-13T16:34:43.206 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_raid6_recov.ko 2024-09-13T16:34:43.206 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_tx.ko 2024-09-13T16:34:43.206 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_xor.ko 2024-09-13T16:34:43.206 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/raid6test.ko 2024-09-13T16:34:43.206 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authenc.ko 2024-09-13T16:34:43.206 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authencesn.ko 2024-09-13T16:34:43.206 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blake2b_generic.ko 2024-09-13T16:34:43.206 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_common.ko 2024-09-13T16:34:43.206 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_generic.ko 2024-09-13T16:34:43.206 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/camellia_generic.ko 2024-09-13T16:34:43.206 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast5_generic.ko 2024-09-13T16:34:43.206 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast6_generic.ko 2024-09-13T16:34:43.206 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast_common.ko 2024-09-13T16:34:43.206 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ccm.ko 2024-09-13T16:34:43.207 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cmac.ko 2024-09-13T16:34:43.207 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crc32_generic.ko 2024-09-13T16:34:43.207 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_engine.ko 2024-09-13T16:34:43.207 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_user.ko 2024-09-13T16:34:43.207 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cts.ko 2024-09-13T16:34:43.207 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/des_generic.ko 2024-09-13T16:34:43.207 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/dh_generic.ko 2024-09-13T16:34:43.207 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecc.ko 2024-09-13T16:34:43.207 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecdh_generic.ko 2024-09-13T16:34:43.207 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/echainiv.ko 2024-09-13T16:34:43.207 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/essiv.ko 2024-09-13T16:34:43.207 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/fcrypt.ko 2024-09-13T16:34:43.207 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/khazad.ko 2024-09-13T16:34:43.208 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4.ko 2024-09-13T16:34:43.208 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4hc.ko 2024-09-13T16:34:43.208 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/md4.ko 2024-09-13T16:34:43.208 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/michael_mic.ko 2024-09-13T16:34:43.208 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcbc.ko 2024-09-13T16:34:43.208 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcrypt.ko 2024-09-13T16:34:43.208 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/rmd160.ko 2024-09-13T16:34:43.208 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/seed.ko 2024-09-13T16:34:43.208 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/serpent_generic.ko 2024-09-13T16:34:43.208 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tcrypt.ko 2024-09-13T16:34:43.208 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tea.ko 2024-09-13T16:34:43.209 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_common.ko 2024-09-13T16:34:43.209 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_generic.ko 2024-09-13T16:34:43.209 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/vmac.ko 2024-09-13T16:34:43.209 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/wp512.ko 2024-09-13T16:34:43.209 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xcbc.ko 2024-09-13T16:34:43.209 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xor.ko 2024-09-13T16:34:43.209 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xxhash_generic.ko 2024-09-13T16:34:43.209 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers 2024-09-13T16:34:43.209 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi 2024-09-13T16:34:43.209 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_ipmi.ko 2024-09-13T16:34:43.209 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_pad.ko 2024-09-13T16:34:43.209 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/ec_sys.ko 2024-09-13T16:34:43.210 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/platform_profile.ko 2024-09-13T16:34:43.210 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbs.ko 2024-09-13T16:34:43.210 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbshc.ko 2024-09-13T16:34:43.210 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/video.ko 2024-09-13T16:34:43.210 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata 2024-09-13T16:34:43.210 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/acard-ahci.ko 2024-09-13T16:34:43.210 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ahci_platform.ko 2024-09-13T16:34:43.210 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ata_generic.ko 2024-09-13T16:34:43.210 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/libahci_platform.ko 2024-09-13T16:34:43.210 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_acpi.ko 2024-09-13T16:34:43.210 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ali.ko 2024-09-13T16:34:43.210 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_amd.ko 2024-09-13T16:34:43.211 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_artop.ko 2024-09-13T16:34:43.211 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atiixp.ko 2024-09-13T16:34:43.211 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atp867x.ko 2024-09-13T16:34:43.211 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd640.ko 2024-09-13T16:34:43.211 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd64x.ko 2024-09-13T16:34:43.211 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cypress.ko 2024-09-13T16:34:43.211 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_efar.ko 2024-09-13T16:34:43.211 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt366.ko 2024-09-13T16:34:43.211 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt37x.ko 2024-09-13T16:34:43.211 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x2n.ko 2024-09-13T16:34:43.211 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x3.ko 2024-09-13T16:34:43.211 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it8213.ko 2024-09-13T16:34:43.211 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it821x.ko 2024-09-13T16:34:43.211 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_jmicron.ko 2024-09-13T16:34:43.212 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_marvell.ko 2024-09-13T16:34:43.212 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_mpiix.ko 2024-09-13T16:34:43.212 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_netcell.ko 2024-09-13T16:34:43.212 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ninja32.ko 2024-09-13T16:34:43.212 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87410.ko 2024-09-13T16:34:43.212 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87415.ko 2024-09-13T16:34:43.212 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_oldpiix.ko 2024-09-13T16:34:43.212 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_opti.ko 2024-09-13T16:34:43.212 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_optidma.ko 2024-09-13T16:34:43.212 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pcmcia.ko 2024-09-13T16:34:43.212 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc2027x.ko 2024-09-13T16:34:43.213 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-09-13T16:34:43.213 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_piccolo.ko 2024-09-13T16:34:43.213 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_rdc.ko 2024-09-13T16:34:43.213 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sch.ko 2024-09-13T16:34:43.213 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_serverworks.ko 2024-09-13T16:34:43.213 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sil680.ko 2024-09-13T16:34:43.213 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sis.ko 2024-09-13T16:34:43.213 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sl82c105.ko 2024-09-13T16:34:43.213 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_triflex.ko 2024-09-13T16:34:43.213 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_via.ko 2024-09-13T16:34:43.213 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pdc_adma.ko 2024-09-13T16:34:43.214 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_inic162x.ko 2024-09-13T16:34:43.214 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_mv.ko 2024-09-13T16:34:43.214 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_nv.ko 2024-09-13T16:34:43.214 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_promise.ko 2024-09-13T16:34:43.214 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_qstor.ko 2024-09-13T16:34:43.214 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil.ko 2024-09-13T16:34:43.214 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil24.ko 2024-09-13T16:34:43.214 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sis.ko 2024-09-13T16:34:43.214 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_svw.ko 2024-09-13T16:34:43.214 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sx4.ko 2024-09-13T16:34:43.214 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_uli.ko 2024-09-13T16:34:43.214 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_via.ko 2024-09-13T16:34:43.214 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_vsc.ko 2024-09-13T16:34:43.215 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay 2024-09-13T16:34:43.215 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay/ks0108.ko 2024-09-13T16:34:43.215 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base 2024-09-13T16:34:43.215 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap 2024-09-13T16:34:43.215 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-i2c.ko 2024-09-13T16:34:43.215 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-mmio.ko 2024-09-13T16:34:43.215 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma 2024-09-13T16:34:43.215 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma/bcma.ko 2024-09-13T16:34:43.215 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block 2024-09-13T16:34:43.215 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe 2024-09-13T16:34:43.215 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe/aoe.ko 2024-09-13T16:34:43.215 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/brd.ko 2024-09-13T16:34:43.216 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd 2024-09-13T16:34:43.216 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd/drbd.ko 2024-09-13T16:34:43.216 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/floppy.ko 2024-09-13T16:34:43.216 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/loop.ko 2024-09-13T16:34:43.216 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx 2024-09-13T16:34:43.216 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-09-13T16:34:43.216 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/nbd.ko 2024-09-13T16:34:43.216 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk 2024-09-13T16:34:43.216 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk/null_blk.ko 2024-09-13T16:34:43.216 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/pktcdvd.ko 2024-09-13T16:34:43.217 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/rbd.ko 2024-09-13T16:34:43.217 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/virtio_blk.ko 2024-09-13T16:34:43.217 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback 2024-09-13T16:34:43.217 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-09-13T16:34:43.217 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkfront.ko 2024-09-13T16:34:43.217 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram 2024-09-13T16:34:43.217 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram/zram.ko 2024-09-13T16:34:43.217 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char 2024-09-13T16:34:43.217 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hangcheck-timer.ko 2024-09-13T16:34:43.217 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random 2024-09-13T16:34:43.217 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/amd-rng.ko 2024-09-13T16:34:43.217 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/intel-rng.ko 2024-09-13T16:34:43.218 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-09-13T16:34:43.218 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/via-rng.ko 2024-09-13T16:34:43.218 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/virtio-rng.ko 2024-09-13T16:34:43.218 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi 2024-09-13T16:34:43.218 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-09-13T16:34:43.218 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-09-13T16:34:43.218 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-09-13T16:34:43.218 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_si.ko 2024-09-13T16:34:43.218 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-09-13T16:34:43.218 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-09-13T16:34:43.218 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/lp.ko 2024-09-13T16:34:43.219 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave 2024-09-13T16:34:43.219 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave/mwave.ko 2024-09-13T16:34:43.219 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ppdev.ko 2024-09-13T16:34:43.219 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tlclk.ko 2024-09-13T16:34:43.219 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm 2024-09-13T16:34:43.219 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm.ko 2024-09-13T16:34:43.219 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_atmel.ko 2024-09-13T16:34:43.219 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_infineon.ko 2024-09-13T16:34:43.219 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_nsc.ko 2024-09-13T16:34:43.219 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis.ko 2024-09-13T16:34:43.219 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-09-13T16:34:43.219 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/uv_mmtimer.ko 2024-09-13T16:34:43.219 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/virtio_console.ko 2024-09-13T16:34:43.219 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq 2024-09-13T16:34:43.220 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-09-13T16:34:43.220 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-09-13T16:34:43.220 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/p4-clockmod.ko 2024-09-13T16:34:43.220 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-09-13T16:34:43.220 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/powernow-k8.ko 2024-09-13T16:34:43.220 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/speedstep-lib.ko 2024-09-13T16:34:43.220 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto 2024-09-13T16:34:43.220 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp 2024-09-13T16:34:43.220 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-09-13T16:34:43.220 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp.ko 2024-09-13T16:34:43.220 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel 2024-09-13T16:34:43.220 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat 2024-09-13T16:34:43.221 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common 2024-09-13T16:34:43.221 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-09-13T16:34:43.221 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-09-13T16:34:43.221 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-09-13T16:34:43.221 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-aes.ko 2024-09-13T16:34:43.221 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-sha.ko 2024-09-13T16:34:43.221 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio 2024-09-13T16:34:43.221 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-09-13T16:34:43.221 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca 2024-09-13T16:34:43.221 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca/dca.ko 2024-09-13T16:34:43.221 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq 2024-09-13T16:34:43.221 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-09-13T16:34:43.221 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma 2024-09-13T16:34:43.222 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw 2024-09-13T16:34:43.222 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw/dw_dmac.ko 2024-09-13T16:34:43.222 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat 2024-09-13T16:34:43.222 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat/ioatdma.ko 2024-09-13T16:34:43.222 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac 2024-09-13T16:34:43.222 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/amd64_edac.ko 2024-09-13T16:34:43.222 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/e752x_edac.ko 2024-09-13T16:34:43.222 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/edac_mce_amd.ko 2024-09-13T16:34:43.222 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3000_edac.ko 2024-09-13T16:34:43.222 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3200_edac.ko 2024-09-13T16:34:43.222 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5100_edac.ko 2024-09-13T16:34:43.222 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5400_edac.ko 2024-09-13T16:34:43.222 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7300_edac.ko 2024-09-13T16:34:43.222 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7core_edac.ko 2024-09-13T16:34:43.223 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i82975x_edac.ko 2024-09-13T16:34:43.223 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/ie31200_edac.ko 2024-09-13T16:34:43.223 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/sb_edac.ko 2024-09-13T16:34:43.223 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/x38_edac.ko 2024-09-13T16:34:43.223 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware 2024-09-13T16:34:43.223 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/edd.ko 2024-09-13T16:34:43.223 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/iscsi_ibft.ko 2024-09-13T16:34:43.223 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio 2024-09-13T16:34:43.223 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-ich.ko 2024-09-13T16:34:43.223 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-viperboard.ko 2024-09-13T16:34:43.223 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid 2024-09-13T16:34:43.223 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-appleir.ko 2024-09-13T16:34:43.223 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-aureal.ko 2024-09-13T16:34:43.224 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-axff.ko 2024-09-13T16:34:43.224 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-dr.ko 2024-09-13T16:34:43.224 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elecom.ko 2024-09-13T16:34:43.224 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elo.ko 2024-09-13T16:34:43.224 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-emsff.ko 2024-09-13T16:34:43.224 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gaff.ko 2024-09-13T16:34:43.224 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gt683r.ko 2024-09-13T16:34:43.224 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gyration.ko 2024-09-13T16:34:43.224 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-kbd.ko 2024-09-13T16:34:43.224 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-mouse.ko 2024-09-13T16:34:43.224 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtekff.ko 2024-09-13T16:34:43.224 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-hyperv.ko 2024-09-13T16:34:43.224 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-icade.ko 2024-09-13T16:34:43.225 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-keytouch.ko 2024-09-13T16:34:43.225 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-kye.ko 2024-09-13T16:34:43.225 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lcpower.ko 2024-09-13T16:34:43.225 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-led.ko 2024-09-13T16:34:43.225 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lenovo.ko 2024-09-13T16:34:43.225 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-dj.ko 2024-09-13T16:34:43.225 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-09-13T16:34:43.225 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-multitouch.ko 2024-09-13T16:34:43.225 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-ortek.ko 2024-09-13T16:34:43.225 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-penmount.ko 2024-09-13T16:34:43.225 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-petalynx.ko 2024-09-13T16:34:43.225 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-picolcd.ko 2024-09-13T16:34:43.226 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-pl.ko 2024-09-13T16:34:43.226 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-plantronics.ko 2024-09-13T16:34:43.226 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-primax.ko 2024-09-13T16:34:43.226 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-rmi.ko 2024-09-13T16:34:43.226 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-arvo.ko 2024-09-13T16:34:43.226 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-common.ko 2024-09-13T16:34:43.226 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-isku.ko 2024-09-13T16:34:43.226 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kone.ko 2024-09-13T16:34:43.226 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-09-13T16:34:43.226 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-konepure.ko 2024-09-13T16:34:43.226 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-09-13T16:34:43.226 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-lua.ko 2024-09-13T16:34:43.226 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-pyra.ko 2024-09-13T16:34:43.227 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-ryos.ko 2024-09-13T16:34:43.227 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-savu.ko 2024-09-13T16:34:43.227 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat.ko 2024-09-13T16:34:43.227 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-saitek.ko 2024-09-13T16:34:43.227 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-samsung.ko 2024-09-13T16:34:43.227 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sensor-hub.ko 2024-09-13T16:34:43.227 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sjoy.ko 2024-09-13T16:34:43.227 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sony.ko 2024-09-13T16:34:43.227 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-speedlink.ko 2024-09-13T16:34:43.227 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-steelseries.ko 2024-09-13T16:34:43.227 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sunplus.ko 2024-09-13T16:34:43.227 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-thrustmaster.ko 2024-09-13T16:34:43.227 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tivo.ko 2024-09-13T16:34:43.227 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tmff.ko 2024-09-13T16:34:43.228 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-topseed.ko 2024-09-13T16:34:43.228 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-twinhan.ko 2024-09-13T16:34:43.228 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-uclogic.ko 2024-09-13T16:34:43.228 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-waltop.ko 2024-09-13T16:34:43.228 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-wiimote.ko 2024-09-13T16:34:43.228 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-xinmo.ko 2024-09-13T16:34:43.228 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zpff.ko 2024-09-13T16:34:43.228 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zydacron.ko 2024-09-13T16:34:43.228 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/uhid.ko 2024-09-13T16:34:43.228 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/wacom.ko 2024-09-13T16:34:43.228 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv 2024-09-13T16:34:43.228 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_balloon.ko 2024-09-13T16:34:43.229 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_utils.ko 2024-09-13T16:34:43.229 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_vmbus.ko 2024-09-13T16:34:43.229 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon 2024-09-13T16:34:43.229 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru.ko 2024-09-13T16:34:43.229 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru3.ko 2024-09-13T16:34:43.229 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/acpi_power_meter.ko 2024-09-13T16:34:43.229 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7414.ko 2024-09-13T16:34:43.230 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7418.ko 2024-09-13T16:34:43.230 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adc128d818.ko 2024-09-13T16:34:43.230 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1025.ko 2024-09-13T16:34:43.230 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1026.ko 2024-09-13T16:34:43.230 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1029.ko 2024-09-13T16:34:43.230 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1031.ko 2024-09-13T16:34:43.230 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm9240.ko 2024-09-13T16:34:43.230 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ads7828.ko 2024-09-13T16:34:43.230 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7410.ko 2024-09-13T16:34:43.230 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7411.ko 2024-09-13T16:34:43.230 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7462.ko 2024-09-13T16:34:43.230 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7470.ko 2024-09-13T16:34:43.231 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7475.ko 2024-09-13T16:34:43.231 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7x10.ko 2024-09-13T16:34:43.231 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/amc6821.ko 2024-09-13T16:34:43.231 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/applesmc.ko 2024-09-13T16:34:43.231 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asb100.ko 2024-09-13T16:34:43.231 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asc7621.ko 2024-09-13T16:34:43.231 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asus_atk0110.ko 2024-09-13T16:34:43.232 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/atxp1.ko 2024-09-13T16:34:43.232 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/coretemp.ko 2024-09-13T16:34:43.232 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-09-13T16:34:43.232 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dme1737.ko 2024-09-13T16:34:43.232 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds1621.ko 2024-09-13T16:34:43.232 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds620.ko 2024-09-13T16:34:43.232 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc1403.ko 2024-09-13T16:34:43.232 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc6w201.ko 2024-09-13T16:34:43.232 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71805f.ko 2024-09-13T16:34:43.232 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71882fg.ko 2024-09-13T16:34:43.232 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f75375s.ko 2024-09-13T16:34:43.233 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fam15h_power.ko 2024-09-13T16:34:43.233 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fschmd.ko 2024-09-13T16:34:43.233 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g760a.ko 2024-09-13T16:34:43.233 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g762.ko 2024-09-13T16:34:43.233 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl518sm.ko 2024-09-13T16:34:43.233 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl520sm.ko 2024-09-13T16:34:43.233 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/hwmon-vid.ko 2024-09-13T16:34:43.233 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5500_temp.ko 2024-09-13T16:34:43.233 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5k_amb.ko 2024-09-13T16:34:43.234 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmaem.ko 2024-09-13T16:34:43.234 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmpex.ko 2024-09-13T16:34:43.234 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina209.ko 2024-09-13T16:34:43.234 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina2xx.ko 2024-09-13T16:34:43.234 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/it87.ko 2024-09-13T16:34:43.234 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k10temp.ko 2024-09-13T16:34:43.234 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k8temp.ko 2024-09-13T16:34:43.234 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lineage-pem.ko 2024-09-13T16:34:43.234 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm63.ko 2024-09-13T16:34:43.234 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm73.ko 2024-09-13T16:34:43.234 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm75.ko 2024-09-13T16:34:43.235 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm77.ko 2024-09-13T16:34:43.235 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm78.ko 2024-09-13T16:34:43.235 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm80.ko 2024-09-13T16:34:43.235 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm83.ko 2024-09-13T16:34:43.235 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm85.ko 2024-09-13T16:34:43.235 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm87.ko 2024-09-13T16:34:43.235 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm90.ko 2024-09-13T16:34:43.235 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm92.ko 2024-09-13T16:34:43.235 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm93.ko 2024-09-13T16:34:43.235 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95234.ko 2024-09-13T16:34:43.235 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95241.ko 2024-09-13T16:34:43.235 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95245.ko 2024-09-13T16:34:43.235 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc2945.ko 2024-09-13T16:34:43.236 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4151.ko 2024-09-13T16:34:43.236 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4215.ko 2024-09-13T16:34:43.236 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4222.ko 2024-09-13T16:34:43.236 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4245.ko 2024-09-13T16:34:43.236 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4260.ko 2024-09-13T16:34:43.236 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4261.ko 2024-09-13T16:34:43.236 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max16065.ko 2024-09-13T16:34:43.236 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1619.ko 2024-09-13T16:34:43.236 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1668.ko 2024-09-13T16:34:43.236 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max197.ko 2024-09-13T16:34:43.236 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6639.ko 2024-09-13T16:34:43.236 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6650.ko 2024-09-13T16:34:43.236 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6697.ko 2024-09-13T16:34:43.237 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/mcp3021.ko 2024-09-13T16:34:43.237 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6683.ko 2024-09-13T16:34:43.237 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775-core.ko 2024-09-13T16:34:43.237 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775.ko 2024-09-13T16:34:43.237 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct7802.ko 2024-09-13T16:34:43.237 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87360.ko 2024-09-13T16:34:43.237 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87427.ko 2024-09-13T16:34:43.237 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pcf8591.ko 2024-09-13T16:34:43.237 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus 2024-09-13T16:34:43.237 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-09-13T16:34:43.237 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-09-13T16:34:43.237 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-09-13T16:34:43.238 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max16064.ko 2024-09-13T16:34:43.238 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max34440.ko 2024-09-13T16:34:43.238 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max8688.ko 2024-09-13T16:34:43.238 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-09-13T16:34:43.238 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-09-13T16:34:43.238 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-09-13T16:34:43.238 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-09-13T16:34:43.238 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-09-13T16:34:43.239 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-09-13T16:34:43.239 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/powr1220.ko 2024-09-13T16:34:43.239 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5627.ko 2024-09-13T16:34:43.239 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5636.ko 2024-09-13T16:34:43.239 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch56xx-common.ko 2024-09-13T16:34:43.239 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht15.ko 2024-09-13T16:34:43.239 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht21.ko 2024-09-13T16:34:43.239 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/shtc1.ko 2024-09-13T16:34:43.239 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sis5595.ko 2024-09-13T16:34:43.239 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47b397.ko 2024-09-13T16:34:43.239 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m1.ko 2024-09-13T16:34:43.239 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m192.ko 2024-09-13T16:34:43.240 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/thmc50.ko 2024-09-13T16:34:43.240 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp102.ko 2024-09-13T16:34:43.240 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp103.ko 2024-09-13T16:34:43.240 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp401.ko 2024-09-13T16:34:43.241 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp421.ko 2024-09-13T16:34:43.241 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via-cputemp.ko 2024-09-13T16:34:43.241 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via686a.ko 2024-09-13T16:34:43.241 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt1211.ko 2024-09-13T16:34:43.241 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt8231.ko 2024-09-13T16:34:43.242 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627ehf.ko 2024-09-13T16:34:43.242 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627hf.ko 2024-09-13T16:34:43.242 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83781d.ko 2024-09-13T16:34:43.242 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83791d.ko 2024-09-13T16:34:43.242 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83792d.ko 2024-09-13T16:34:43.242 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83793.ko 2024-09-13T16:34:43.242 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83795.ko 2024-09-13T16:34:43.242 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l785ts.ko 2024-09-13T16:34:43.243 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l786ng.ko 2024-09-13T16:34:43.243 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c 2024-09-13T16:34:43.243 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos 2024-09-13T16:34:43.243 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-09-13T16:34:43.243 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-09-13T16:34:43.243 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses 2024-09-13T16:34:43.243 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-09-13T16:34:43.243 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-09-13T16:34:43.243 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-09-13T16:34:43.244 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-09-13T16:34:43.244 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-09-13T16:34:43.244 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-09-13T16:34:43.244 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-09-13T16:34:43.244 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-09-13T16:34:43.244 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-i801.ko 2024-09-13T16:34:43.244 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-isch.ko 2024-09-13T16:34:43.244 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-09-13T16:34:43.245 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-09-13T16:34:43.245 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-09-13T16:34:43.245 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-parport.ko 2024-09-13T16:34:43.245 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-09-13T16:34:43.245 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-09-13T16:34:43.245 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-09-13T16:34:43.245 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-09-13T16:34:43.245 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-09-13T16:34:43.245 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-09-13T16:34:43.245 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-via.ko 2024-09-13T16:34:43.245 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-09-13T16:34:43.246 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-09-13T16:34:43.246 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-dev.ko 2024-09-13T16:34:43.246 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-mux.ko 2024-09-13T16:34:43.246 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-09-13T16:34:43.246 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-smbus.ko 2024-09-13T16:34:43.246 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-stub.ko 2024-09-13T16:34:43.246 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband 2024-09-13T16:34:43.246 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core 2024-09-13T16:34:43.246 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_cm.ko 2024-09-13T16:34:43.247 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_core.ko 2024-09-13T16:34:43.247 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_umad.ko 2024-09-13T16:34:43.247 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-09-13T16:34:43.247 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/iw_cm.ko 2024-09-13T16:34:43.247 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_cm.ko 2024-09-13T16:34:43.247 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-09-13T16:34:43.247 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw 2024-09-13T16:34:43.247 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4 2024-09-13T16:34:43.247 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-09-13T16:34:43.247 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4 2024-09-13T16:34:43.247 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-09-13T16:34:43.247 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5 2024-09-13T16:34:43.247 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-09-13T16:34:43.247 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca 2024-09-13T16:34:43.248 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-09-13T16:34:43.248 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp 2024-09-13T16:34:43.248 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib 2024-09-13T16:34:43.248 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-09-13T16:34:43.248 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser 2024-09-13T16:34:43.248 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-09-13T16:34:43.248 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert 2024-09-13T16:34:43.248 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-09-13T16:34:43.248 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp 2024-09-13T16:34:43.248 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-09-13T16:34:43.248 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt 2024-09-13T16:34:43.249 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-09-13T16:34:43.249 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input 2024-09-13T16:34:43.249 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport 2024-09-13T16:34:43.249 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-09-13T16:34:43.249 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/fm801-gp.ko 2024-09-13T16:34:43.249 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/gameport.ko 2024-09-13T16:34:43.249 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/joydev.ko 2024-09-13T16:34:43.249 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard 2024-09-13T16:34:43.249 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard/gpio_keys.ko 2024-09-13T16:34:43.249 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse 2024-09-13T16:34:43.249 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/appletouch.ko 2024-09-13T16:34:43.249 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/bcm5974.ko 2024-09-13T16:34:43.249 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/cyapatp.ko 2024-09-13T16:34:43.250 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/elan_i2c.ko 2024-09-13T16:34:43.250 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/sermouse.ko 2024-09-13T16:34:43.250 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-09-13T16:34:43.250 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_usb.ko 2024-09-13T16:34:43.250 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/vsxxxaa.ko 2024-09-13T16:34:43.250 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4 2024-09-13T16:34:43.250 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4/rmi_core.ko 2024-09-13T16:34:43.250 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio 2024-09-13T16:34:43.251 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/altera_ps2.ko 2024-09-13T16:34:43.251 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/arc_ps2.ko 2024-09-13T16:34:43.251 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-09-13T16:34:43.251 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/serio_raw.ko 2024-09-13T16:34:43.251 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/sparse-keymap.ko 2024-09-13T16:34:43.251 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds 2024-09-13T16:34:43.251 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-blinkm.ko 2024-09-13T16:34:43.251 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lm3530.ko 2024-09-13T16:34:43.251 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lp3944.ko 2024-09-13T16:34:43.251 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-ss4200.ko 2024-09-13T16:34:43.251 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger 2024-09-13T16:34:43.251 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-09-13T16:34:43.252 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-09-13T16:34:43.252 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-09-13T16:34:43.252 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-09-13T16:34:43.252 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-09-13T16:34:43.252 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-09-13T16:34:43.252 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-09-13T16:34:43.252 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-09-13T16:34:43.252 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md 2024-09-13T16:34:43.252 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache 2024-09-13T16:34:43.252 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache/bcache.ko 2024-09-13T16:34:43.252 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-bio-prison.ko 2024-09-13T16:34:43.252 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache-smq.ko 2024-09-13T16:34:43.253 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache.ko 2024-09-13T16:34:43.253 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-crypt.ko 2024-09-13T16:34:43.253 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-delay.ko 2024-09-13T16:34:43.253 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-flakey.ko 2024-09-13T16:34:43.253 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-log-userspace.ko 2024-09-13T16:34:43.253 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-multipath.ko 2024-09-13T16:34:43.253 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-queue-length.ko 2024-09-13T16:34:43.253 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-raid.ko 2024-09-13T16:34:43.253 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-round-robin.ko 2024-09-13T16:34:43.253 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-service-time.ko 2024-09-13T16:34:43.253 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-switch.ko 2024-09-13T16:34:43.254 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-thin-pool.ko 2024-09-13T16:34:43.254 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-verity.ko 2024-09-13T16:34:43.254 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data 2024-09-13T16:34:43.254 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-09-13T16:34:43.254 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid0.ko 2024-09-13T16:34:43.254 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid1.ko 2024-09-13T16:34:43.254 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid10.ko 2024-09-13T16:34:43.254 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid456.ko 2024-09-13T16:34:43.254 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message 2024-09-13T16:34:43.254 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion 2024-09-13T16:34:43.254 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptbase.ko 2024-09-13T16:34:43.254 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptctl.ko 2024-09-13T16:34:43.254 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptfc.ko 2024-09-13T16:34:43.254 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptlan.ko 2024-09-13T16:34:43.255 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptsas.ko 2024-09-13T16:34:43.255 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptscsih.ko 2024-09-13T16:34:43.255 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptspi.ko 2024-09-13T16:34:43.255 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd 2024-09-13T16:34:43.255 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_ich.ko 2024-09-13T16:34:43.255 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_sch.ko 2024-09-13T16:34:43.255 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/mfd-core.ko 2024-09-13T16:34:43.255 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/sm501.ko 2024-09-13T16:34:43.256 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/viperboard.ko 2024-09-13T16:34:43.256 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/vx855.ko 2024-09-13T16:34:43.256 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/wl1273-core.ko 2024-09-13T16:34:43.256 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc 2024-09-13T16:34:43.256 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl 2024-09-13T16:34:43.256 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-09-13T16:34:43.256 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds9802als.ko 2024-09-13T16:34:43.256 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds990x.ko 2024-09-13T16:34:43.256 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/bh1770glc.ko 2024-09-13T16:34:43.256 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710 2024-09-13T16:34:43.256 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710/cb710.ko 2024-09-13T16:34:43.256 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom 2024-09-13T16:34:43.256 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/at24.ko 2024-09-13T16:34:43.256 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-09-13T16:34:43.257 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/max6875.ko 2024-09-13T16:34:43.257 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/enclosure.ko 2024-09-13T16:34:43.257 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/hpilo.ko 2024-09-13T16:34:43.257 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29003.ko 2024-09-13T16:34:43.257 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29020.ko 2024-09-13T16:34:43.257 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d 2024-09-13T16:34:43.257 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-09-13T16:34:43.257 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-09-13T16:34:43.257 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei 2024-09-13T16:34:43.257 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-me.ko 2024-09-13T16:34:43.257 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-txe.ko 2024-09-13T16:34:43.257 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei.ko 2024-09-13T16:34:43.257 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru 2024-09-13T16:34:43.258 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru/gru.ko 2024-09-13T16:34:43.258 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp 2024-09-13T16:34:43.258 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xp.ko 2024-09-13T16:34:43.258 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpc.ko 2024-09-13T16:34:43.258 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-09-13T16:34:43.258 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_7xx1.ko 2024-09-13T16:34:43.258 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_core.ko 2024-09-13T16:34:43.258 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tsl2550.ko 2024-09-13T16:34:43.258 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_balloon.ko 2024-09-13T16:34:43.258 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci 2024-09-13T16:34:43.258 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-09-13T16:34:43.258 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net 2024-09-13T16:34:43.258 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding 2024-09-13T16:34:43.258 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding/bonding.ko 2024-09-13T16:34:43.259 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa 2024-09-13T16:34:43.259 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53 2024-09-13T16:34:43.259 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53/b53_common.ko 2024-09-13T16:34:43.259 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/bcm-sf2.ko 2024-09-13T16:34:43.259 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6060.ko 2024-09-13T16:34:43.259 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx 2024-09-13T16:34:43.259 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-09-13T16:34:43.259 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dummy.ko 2024-09-13T16:34:43.259 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/eql.ko 2024-09-13T16:34:43.259 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet 2024-09-13T16:34:43.259 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com 2024-09-13T16:34:43.259 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-09-13T16:34:43.259 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-09-13T16:34:43.259 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-09-13T16:34:43.260 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-09-13T16:34:43.260 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390 2024-09-13T16:34:43.260 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/8390.ko 2024-09-13T16:34:43.260 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-09-13T16:34:43.260 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-09-13T16:34:43.260 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-09-13T16:34:43.260 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec 2024-09-13T16:34:43.260 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-09-13T16:34:43.260 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere 2024-09-13T16:34:43.260 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere/et131x.ko 2024-09-13T16:34:43.260 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon 2024-09-13T16:34:43.260 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-09-13T16:34:43.260 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera 2024-09-13T16:34:43.261 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-09-13T16:34:43.261 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd 2024-09-13T16:34:43.261 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-09-13T16:34:43.261 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-09-13T16:34:43.261 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-09-13T16:34:43.261 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros 2024-09-13T16:34:43.261 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx 2024-09-13T16:34:43.261 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-09-13T16:34:43.261 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c 2024-09-13T16:34:43.261 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-09-13T16:34:43.261 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e 2024-09-13T16:34:43.261 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-09-13T16:34:43.261 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx 2024-09-13T16:34:43.261 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-09-13T16:34:43.262 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-09-13T16:34:43.262 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom 2024-09-13T16:34:43.262 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-09-13T16:34:43.262 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-09-13T16:34:43.262 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-09-13T16:34:43.262 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-09-13T16:34:43.262 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-09-13T16:34:43.262 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet 2024-09-13T16:34:43.262 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-09-13T16:34:43.262 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-09-13T16:34:43.262 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade 2024-09-13T16:34:43.262 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna 2024-09-13T16:34:43.262 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-09-13T16:34:43.262 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio 2024-09-13T16:34:43.263 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb 2024-09-13T16:34:43.263 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-09-13T16:34:43.263 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-09-13T16:34:43.263 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-09-13T16:34:43.263 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-09-13T16:34:43.263 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-09-13T16:34:43.263 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-09-13T16:34:43.263 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-09-13T16:34:43.263 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-09-13T16:34:43.263 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-09-13T16:34:43.263 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco 2024-09-13T16:34:43.263 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic 2024-09-13T16:34:43.263 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-09-13T16:34:43.263 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec 2024-09-13T16:34:43.263 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip 2024-09-13T16:34:43.264 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-09-13T16:34:43.264 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-09-13T16:34:43.264 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-09-13T16:34:43.264 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-09-13T16:34:43.264 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-09-13T16:34:43.264 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-09-13T16:34:43.264 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink 2024-09-13T16:34:43.264 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-09-13T16:34:43.264 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-09-13T16:34:43.264 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dnet.ko 2024-09-13T16:34:43.264 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex 2024-09-13T16:34:43.264 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet 2024-09-13T16:34:43.264 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-09-13T16:34:43.264 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ethoc.ko 2024-09-13T16:34:43.265 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/fealnx.ko 2024-09-13T16:34:43.265 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel 2024-09-13T16:34:43.265 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e100.ko 2024-09-13T16:34:43.265 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000 2024-09-13T16:34:43.265 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-09-13T16:34:43.265 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e 2024-09-13T16:34:43.265 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-09-13T16:34:43.266 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k 2024-09-13T16:34:43.266 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-09-13T16:34:43.266 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e 2024-09-13T16:34:43.266 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-09-13T16:34:43.266 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf 2024-09-13T16:34:43.266 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-09-13T16:34:43.266 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb 2024-09-13T16:34:43.266 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-09-13T16:34:43.266 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf 2024-09-13T16:34:43.266 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-09-13T16:34:43.266 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe 2024-09-13T16:34:43.267 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-09-13T16:34:43.267 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf 2024-09-13T16:34:43.267 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-09-13T16:34:43.267 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth 2024-09-13T16:34:43.267 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-09-13T16:34:43.267 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie 2024-09-13T16:34:43.267 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-09-13T16:34:43.267 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/jme.ko 2024-09-13T16:34:43.267 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell 2024-09-13T16:34:43.267 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-09-13T16:34:43.268 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/skge.ko 2024-09-13T16:34:43.268 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-09-13T16:34:43.268 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox 2024-09-13T16:34:43.268 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4 2024-09-13T16:34:43.268 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-09-13T16:34:43.268 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-09-13T16:34:43.268 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5 2024-09-13T16:34:43.268 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-09-13T16:34:43.268 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-09-13T16:34:43.269 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel 2024-09-13T16:34:43.269 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-09-13T16:34:43.269 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom 2024-09-13T16:34:43.269 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge 2024-09-13T16:34:43.269 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-09-13T16:34:43.269 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi 2024-09-13T16:34:43.269 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-09-13T16:34:43.269 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-09-13T16:34:43.269 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion 2024-09-13T16:34:43.269 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-09-13T16:34:43.270 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia 2024-09-13T16:34:43.270 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-09-13T16:34:43.270 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines 2024-09-13T16:34:43.270 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-09-13T16:34:43.270 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-09-13T16:34:43.270 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic 2024-09-13T16:34:43.270 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen 2024-09-13T16:34:43.270 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-09-13T16:34:43.270 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-09-13T16:34:43.270 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-09-13T16:34:43.270 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-09-13T16:34:43.270 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc 2024-09-13T16:34:43.271 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-09-13T16:34:43.271 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek 2024-09-13T16:34:43.271 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-09-13T16:34:43.271 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-09-13T16:34:43.271 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/atp.ko 2024-09-13T16:34:43.271 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-09-13T16:34:43.271 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker 2024-09-13T16:34:43.271 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-09-13T16:34:43.271 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc 2024-09-13T16:34:43.271 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-09-13T16:34:43.271 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan 2024-09-13T16:34:43.271 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-09-13T16:34:43.272 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis 2024-09-13T16:34:43.272 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis190.ko 2024-09-13T16:34:43.272 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis900.ko 2024-09-13T16:34:43.272 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc 2024-09-13T16:34:43.272 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-09-13T16:34:43.272 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-09-13T16:34:43.272 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-09-13T16:34:43.272 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-09-13T16:34:43.272 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro 2024-09-13T16:34:43.272 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac 2024-09-13T16:34:43.272 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-09-13T16:34:43.272 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-09-13T16:34:43.273 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun 2024-09-13T16:34:43.273 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/cassini.ko 2024-09-13T16:34:43.273 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/niu.ko 2024-09-13T16:34:43.273 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sungem.ko 2024-09-13T16:34:43.273 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-09-13T16:34:43.273 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti 2024-09-13T16:34:43.273 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-09-13T16:34:43.273 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti 2024-09-13T16:34:43.273 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti/tlan.ko 2024-09-13T16:34:43.273 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via 2024-09-13T16:34:43.273 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-09-13T16:34:43.274 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-09-13T16:34:43.274 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet 2024-09-13T16:34:43.274 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-09-13T16:34:43.274 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-09-13T16:34:43.274 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom 2024-09-13T16:34:43.274 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-09-13T16:34:43.274 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/geneve.ko 2024-09-13T16:34:43.274 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv 2024-09-13T16:34:43.274 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-09-13T16:34:43.274 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154 2024-09-13T16:34:43.274 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154/fakelb.ko 2024-09-13T16:34:43.274 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ifb.ko 2024-09-13T16:34:43.274 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan 2024-09-13T16:34:43.275 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan/ipvlan.ko 2024-09-13T16:34:43.275 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvlan.ko 2024-09-13T16:34:43.275 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvtap.ko 2024-09-13T16:34:43.275 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio 2024-09-13T16:34:43.275 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio.ko 2024-09-13T16:34:43.275 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-09-13T16:34:43.275 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-09-13T16:34:43.275 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-regmap.ko 2024-09-13T16:34:43.275 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mii.ko 2024-09-13T16:34:43.275 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/net_failover.ko 2024-09-13T16:34:43.275 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/netconsole.ko 2024-09-13T16:34:43.275 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/nlmon.ko 2024-09-13T16:34:43.275 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs 2024-09-13T16:34:43.275 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs-lynx.ko 2024-09-13T16:34:43.276 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-09-13T16:34:43.276 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy 2024-09-13T16:34:43.276 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/amd.ko 2024-09-13T16:34:43.276 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-09-13T16:34:43.276 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-09-13T16:34:43.276 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm7xxx.ko 2024-09-13T16:34:43.276 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm87xx.ko 2024-09-13T16:34:43.276 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/broadcom.ko 2024-09-13T16:34:43.276 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/cicada.ko 2024-09-13T16:34:43.276 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/davicom.ko 2024-09-13T16:34:43.276 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/dp83640.ko 2024-09-13T16:34:43.276 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/et1011c.ko 2024-09-13T16:34:43.276 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/icplus.ko 2024-09-13T16:34:43.277 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/lxt.ko 2024-09-13T16:34:43.277 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/marvell.ko 2024-09-13T16:34:43.277 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/micrel.ko 2024-09-13T16:34:43.277 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/national.ko 2024-09-13T16:34:43.277 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/phylink.ko 2024-09-13T16:34:43.277 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/qsemi.ko 2024-09-13T16:34:43.277 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/realtek.ko 2024-09-13T16:34:43.277 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/smsc.ko 2024-09-13T16:34:43.277 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/ste10Xp.ko 2024-09-13T16:34:43.277 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/vitesse.ko 2024-09-13T16:34:43.277 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp 2024-09-13T16:34:43.277 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/bsd_comp.ko 2024-09-13T16:34:43.277 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_async.ko 2024-09-13T16:34:43.278 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_deflate.ko 2024-09-13T16:34:43.278 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_generic.ko 2024-09-13T16:34:43.278 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_mppe.ko 2024-09-13T16:34:43.278 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_synctty.ko 2024-09-13T16:34:43.278 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppoe.ko 2024-09-13T16:34:43.278 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppox.ko 2024-09-13T16:34:43.278 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pptp.ko 2024-09-13T16:34:43.278 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip 2024-09-13T16:34:43.278 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slhc.ko 2024-09-13T16:34:43.278 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slip.ko 2024-09-13T16:34:43.278 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/sungem_phy.ko 2024-09-13T16:34:43.278 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tap.ko 2024-09-13T16:34:43.278 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team 2024-09-13T16:34:43.279 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team.ko 2024-09-13T16:34:43.279 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_activebackup.ko 2024-09-13T16:34:43.279 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_broadcast.ko 2024-09-13T16:34:43.279 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-09-13T16:34:43.279 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_random.ko 2024-09-13T16:34:43.279 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-09-13T16:34:43.279 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tun.ko 2024-09-13T16:34:43.279 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/veth.ko 2024-09-13T16:34:43.279 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/virtio_net.ko 2024-09-13T16:34:43.279 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3 2024-09-13T16:34:43.279 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-09-13T16:34:43.280 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan 2024-09-13T16:34:43.280 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan/vxlan.ko 2024-09-13T16:34:43.280 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback 2024-09-13T16:34:43.280 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback/xen-netback.ko 2024-09-13T16:34:43.280 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netfront.ko 2024-09-13T16:34:43.280 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb 2024-09-13T16:34:43.280 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb/ntb.ko 2024-09-13T16:34:43.280 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme 2024-09-13T16:34:43.280 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host 2024-09-13T16:34:43.280 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme-core.ko 2024-09-13T16:34:43.280 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme.ko 2024-09-13T16:34:43.280 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport 2024-09-13T16:34:43.280 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport.ko 2024-09-13T16:34:43.280 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_cs.ko 2024-09-13T16:34:43.281 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_pc.ko 2024-09-13T16:34:43.281 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_serial.ko 2024-09-13T16:34:43.281 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci 2024-09-13T16:34:43.281 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug 2024-09-13T16:34:43.281 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-09-13T16:34:43.281 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie 2024-09-13T16:34:43.281 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie/aer_inject.ko 2024-09-13T16:34:43.281 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/xen-pcifront.ko 2024-09-13T16:34:43.281 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia 2024-09-13T16:34:43.281 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/i82092.ko 2024-09-13T16:34:43.281 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/pd6729.ko 2024-09-13T16:34:43.282 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/yenta_socket.ko 2024-09-13T16:34:43.282 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl 2024-09-13T16:34:43.282 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel 2024-09-13T16:34:43.282 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-09-13T16:34:43.282 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform 2024-09-13T16:34:43.282 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86 2024-09-13T16:34:43.282 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acer-wmi.ko 2024-09-13T16:34:43.282 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acerhdf.ko 2024-09-13T16:34:43.282 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/apple-gmux.ko 2024-09-13T16:34:43.282 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-laptop.ko 2024-09-13T16:34:43.282 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-09-13T16:34:43.282 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-wmi.ko 2024-09-13T16:34:43.282 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/classmate-laptop.ko 2024-09-13T16:34:43.282 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-09-13T16:34:43.283 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-09-13T16:34:43.283 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-09-13T16:34:43.283 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-09-13T16:34:43.283 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/hdaps.ko 2024-09-13T16:34:43.283 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel 2024-09-13T16:34:43.283 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-09-13T16:34:43.283 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel_ips.ko 2024-09-13T16:34:43.283 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/msi-wmi.ko 2024-09-13T16:34:43.283 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/mxm-wmi.ko 2024-09-13T16:34:43.283 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-09-13T16:34:43.284 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-laptop.ko 2024-09-13T16:34:43.284 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-q10.ko 2024-09-13T16:34:43.284 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/topstar-laptop.ko 2024-09-13T16:34:43.284 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-09-13T16:34:43.284 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_haps.ko 2024-09-13T16:34:43.284 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi-bmof.ko 2024-09-13T16:34:43.284 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi.ko 2024-09-13T16:34:43.284 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap 2024-09-13T16:34:43.284 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_common.ko 2024-09-13T16:34:43.284 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_msr.ko 2024-09-13T16:34:43.284 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps 2024-09-13T16:34:43.285 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients 2024-09-13T16:34:43.285 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-gpio.ko 2024-09-13T16:34:43.285 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-ldisc.ko 2024-09-13T16:34:43.285 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps_parport.ko 2024-09-13T16:34:43.285 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/pps_core.ko 2024-09-13T16:34:43.285 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp 2024-09-13T16:34:43.285 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp.ko 2024-09-13T16:34:43.285 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp_kvm.ko 2024-09-13T16:34:43.285 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras 2024-09-13T16:34:43.285 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd 2024-09-13T16:34:43.285 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl 2024-09-13T16:34:43.285 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-09-13T16:34:43.285 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/fmpm.ko 2024-09-13T16:34:43.286 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc 2024-09-13T16:34:43.286 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-bq32k.ko 2024-09-13T16:34:43.286 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1286.ko 2024-09-13T16:34:43.286 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1307.ko 2024-09-13T16:34:43.286 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1374.ko 2024-09-13T16:34:43.286 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1511.ko 2024-09-13T16:34:43.286 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1553.ko 2024-09-13T16:34:43.286 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1672.ko 2024-09-13T16:34:43.286 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1742.ko 2024-09-13T16:34:43.286 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds2404.ko 2024-09-13T16:34:43.287 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds3232.ko 2024-09-13T16:34:43.287 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-em3027.ko 2024-09-13T16:34:43.287 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-fm3130.ko 2024-09-13T16:34:43.287 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl12022.ko 2024-09-13T16:34:43.287 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl1208.ko 2024-09-13T16:34:43.287 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m41t80.ko 2024-09-13T16:34:43.287 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t35.ko 2024-09-13T16:34:43.287 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t59.ko 2024-09-13T16:34:43.287 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-max6900.ko 2024-09-13T16:34:43.287 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-msm6242.ko 2024-09-13T16:34:43.287 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf2127.ko 2024-09-13T16:34:43.287 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf85063.ko 2024-09-13T16:34:43.287 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8523.ko 2024-09-13T16:34:43.287 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8563.ko 2024-09-13T16:34:43.288 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8583.ko 2024-09-13T16:34:43.288 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rp5c01.ko 2024-09-13T16:34:43.288 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rs5c372.ko 2024-09-13T16:34:43.288 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-09-13T16:34:43.288 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8025.ko 2024-09-13T16:34:43.288 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8581.ko 2024-09-13T16:34:43.288 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-09-13T16:34:43.288 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-x1205.ko 2024-09-13T16:34:43.288 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi 2024-09-13T16:34:43.288 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-9xxx.ko 2024-09-13T16:34:43.288 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-sas.ko 2024-09-13T16:34:43.288 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-xxxx.ko 2024-09-13T16:34:43.289 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/BusLogic.ko 2024-09-13T16:34:43.289 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/a100u2w.ko 2024-09-13T16:34:43.289 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid 2024-09-13T16:34:43.289 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid/aacraid.ko 2024-09-13T16:34:43.289 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/advansys.ko 2024-09-13T16:34:43.289 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx 2024-09-13T16:34:43.289 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-09-13T16:34:43.289 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-09-13T16:34:43.289 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/am53c974.ko 2024-09-13T16:34:43.289 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr 2024-09-13T16:34:43.289 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-09-13T16:34:43.289 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/atp870u.ko 2024-09-13T16:34:43.290 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi 2024-09-13T16:34:43.290 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-09-13T16:34:43.290 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa 2024-09-13T16:34:43.290 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa/bfa.ko 2024-09-13T16:34:43.290 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc 2024-09-13T16:34:43.290 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-09-13T16:34:43.290 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i 2024-09-13T16:34:43.290 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-09-13T16:34:43.290 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ch.ko 2024-09-13T16:34:43.290 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor 2024-09-13T16:34:43.290 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor/csiostor.ko 2024-09-13T16:34:43.290 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi 2024-09-13T16:34:43.290 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i 2024-09-13T16:34:43.291 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-09-13T16:34:43.291 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i 2024-09-13T16:34:43.291 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-09-13T16:34:43.291 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-09-13T16:34:43.291 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dc395x.ko 2024-09-13T16:34:43.291 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler 2024-09-13T16:34:43.291 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-09-13T16:34:43.291 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-09-13T16:34:43.291 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-09-13T16:34:43.291 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-09-13T16:34:43.291 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dmx3191d.ko 2024-09-13T16:34:43.291 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r 2024-09-13T16:34:43.291 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r/esas2r.ko 2024-09-13T16:34:43.291 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esp_scsi.ko 2024-09-13T16:34:43.291 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe 2024-09-13T16:34:43.292 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/fcoe.ko 2024-09-13T16:34:43.292 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-09-13T16:34:43.292 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic 2024-09-13T16:34:43.292 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic/fnic.ko 2024-09-13T16:34:43.292 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hpsa.ko 2024-09-13T16:34:43.292 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hptiop.ko 2024-09-13T16:34:43.292 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hv_storvsc.ko 2024-09-13T16:34:43.292 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/initio.ko 2024-09-13T16:34:43.292 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ipr.ko 2024-09-13T16:34:43.292 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ips.ko 2024-09-13T16:34:43.292 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci 2024-09-13T16:34:43.292 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci/isci.ko 2024-09-13T16:34:43.292 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-09-13T16:34:43.293 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_tcp.ko 2024-09-13T16:34:43.293 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc 2024-09-13T16:34:43.293 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc/libfc.ko 2024-09-13T16:34:43.293 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi.ko 2024-09-13T16:34:43.293 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi_tcp.ko 2024-09-13T16:34:43.293 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas 2024-09-13T16:34:43.293 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas/libsas.ko 2024-09-13T16:34:43.293 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc 2024-09-13T16:34:43.293 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc/lpfc.ko 2024-09-13T16:34:43.293 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid 2024-09-13T16:34:43.294 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid.ko 2024-09-13T16:34:43.294 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-09-13T16:34:43.294 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-09-13T16:34:43.294 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-09-13T16:34:43.294 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas 2024-09-13T16:34:43.294 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-09-13T16:34:43.294 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas 2024-09-13T16:34:43.294 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas/mvsas.ko 2024-09-13T16:34:43.294 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvumi.ko 2024-09-13T16:34:43.294 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001 2024-09-13T16:34:43.294 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-09-13T16:34:43.294 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pmcraid.ko 2024-09-13T16:34:43.294 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla1280.ko 2024-09-13T16:34:43.294 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx 2024-09-13T16:34:43.295 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-09-13T16:34:43.295 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-09-13T16:34:43.295 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx 2024-09-13T16:34:43.295 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-09-13T16:34:43.295 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/raid_class.ko 2024-09-13T16:34:43.295 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_debug.ko 2024-09-13T16:34:43.295 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_fc.ko 2024-09-13T16:34:43.295 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-09-13T16:34:43.295 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_sas.ko 2024-09-13T16:34:43.295 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_spi.ko 2024-09-13T16:34:43.295 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_srp.ko 2024-09-13T16:34:43.295 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ses.ko 2024-09-13T16:34:43.295 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/st.ko 2024-09-13T16:34:43.295 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/stex.ko 2024-09-13T16:34:43.296 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2 2024-09-13T16:34:43.296 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-09-13T16:34:43.296 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/virtio_scsi.ko 2024-09-13T16:34:43.296 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/vmw_pvscsi.ko 2024-09-13T16:34:43.296 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/wd719x.ko 2024-09-13T16:34:43.296 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb 2024-09-13T16:34:43.296 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb/ssb.ko 2024-09-13T16:34:43.296 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target 2024-09-13T16:34:43.296 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi 2024-09-13T16:34:43.296 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-09-13T16:34:43.296 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback 2024-09-13T16:34:43.296 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback/tcm_loop.ko 2024-09-13T16:34:43.296 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_file.ko 2024-09-13T16:34:43.296 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_iblock.ko 2024-09-13T16:34:43.297 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_mod.ko 2024-09-13T16:34:43.297 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_pscsi.ko 2024-09-13T16:34:43.297 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_user.ko 2024-09-13T16:34:43.297 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc 2024-09-13T16:34:43.297 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-09-13T16:34:43.297 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal 2024-09-13T16:34:43.297 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel 2024-09-13T16:34:43.297 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal 2024-09-13T16:34:43.297 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-09-13T16:34:43.297 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-09-13T16:34:43.297 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-09-13T16:34:43.297 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-09-13T16:34:43.298 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-09-13T16:34:43.298 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-09-13T16:34:43.298 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-09-13T16:34:43.298 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-09-13T16:34:43.298 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-09-13T16:34:43.298 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-09-13T16:34:43.298 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-09-13T16:34:43.298 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-09-13T16:34:43.298 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-09-13T16:34:43.298 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-09-13T16:34:43.298 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-09-13T16:34:43.298 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-09-13T16:34:43.299 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-09-13T16:34:43.299 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-09-13T16:34:43.299 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty 2024-09-13T16:34:43.299 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless 2024-09-13T16:34:43.299 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-09-13T16:34:43.299 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_gsm.ko 2024-09-13T16:34:43.299 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_hdlc.ko 2024-09-13T16:34:43.299 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/nozomi.ko 2024-09-13T16:34:43.299 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial 2024-09-13T16:34:43.299 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250 2024-09-13T16:34:43.299 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-09-13T16:34:43.299 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/arc_uart.ko 2024-09-13T16:34:43.299 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm 2024-09-13T16:34:43.300 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm/jsm.ko 2024-09-13T16:34:43.300 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs 2024-09-13T16:34:43.300 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core 2024-09-13T16:34:43.300 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core/ufshcd-core.ko 2024-09-13T16:34:43.300 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host 2024-09-13T16:34:43.300 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-09-13T16:34:43.300 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio 2024-09-13T16:34:43.300 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio.ko 2024-09-13T16:34:43.300 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_aec.ko 2024-09-13T16:34:43.300 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_cif.ko 2024-09-13T16:34:43.300 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_pci_generic.ko 2024-09-13T16:34:43.301 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_sercos3.ko 2024-09-13T16:34:43.301 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb 2024-09-13T16:34:43.301 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class 2024-09-13T16:34:43.301 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-acm.ko 2024-09-13T16:34:43.301 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-wdm.ko 2024-09-13T16:34:43.301 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usblp.ko 2024-09-13T16:34:43.301 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usbtmc.ko 2024-09-13T16:34:43.301 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host 2024-09-13T16:34:43.301 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host/sl811-hcd.ko 2024-09-13T16:34:43.301 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image 2024-09-13T16:34:43.301 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/mdc800.ko 2024-09-13T16:34:43.301 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/microtek.ko 2024-09-13T16:34:43.301 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc 2024-09-13T16:34:43.302 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/adutux.ko 2024-09-13T16:34:43.302 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/appledisplay.ko 2024-09-13T16:34:43.302 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi26.ko 2024-09-13T16:34:43.302 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi62.ko 2024-09-13T16:34:43.302 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ezusb.ko 2024-09-13T16:34:43.302 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/idmouse.ko 2024-09-13T16:34:43.302 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/iowarrior.ko 2024-09-13T16:34:43.302 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/isight_firmware.ko 2024-09-13T16:34:43.302 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ldusb.ko 2024-09-13T16:34:43.302 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/legousbtower.ko 2024-09-13T16:34:43.302 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga 2024-09-13T16:34:43.302 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-09-13T16:34:43.303 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/trancevibrator.ko 2024-09-13T16:34:43.303 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usb3503.ko 2024-09-13T16:34:43.303 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usblcd.ko 2024-09-13T16:34:43.303 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usbsevseg.ko 2024-09-13T16:34:43.303 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/uss720.ko 2024-09-13T16:34:43.303 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/yurex.ko 2024-09-13T16:34:43.303 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy 2024-09-13T16:34:43.303 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy/phy-generic.ko 2024-09-13T16:34:43.303 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial 2024-09-13T16:34:43.303 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/aircable.ko 2024-09-13T16:34:43.303 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ark3116.ko 2024-09-13T16:34:43.303 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/belkin_sa.ko 2024-09-13T16:34:43.304 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ch341.ko 2024-09-13T16:34:43.304 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cp210x.ko 2024-09-13T16:34:43.304 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cyberjack.ko 2024-09-13T16:34:43.304 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cypress_m8.ko 2024-09-13T16:34:43.304 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/digi_acceleport.ko 2024-09-13T16:34:43.304 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/empeg.ko 2024-09-13T16:34:43.304 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ftdi_sio.ko 2024-09-13T16:34:43.304 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/garmin_gps.ko 2024-09-13T16:34:43.304 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_edgeport.ko 2024-09-13T16:34:43.304 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_ti.ko 2024-09-13T16:34:43.304 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipaq.ko 2024-09-13T16:34:43.305 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipw.ko 2024-09-13T16:34:43.305 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ir-usb.ko 2024-09-13T16:34:43.305 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-09-13T16:34:43.305 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan.ko 2024-09-13T16:34:43.305 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan_pda.ko 2024-09-13T16:34:43.305 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kl5kusb105.ko 2024-09-13T16:34:43.305 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kobil_sct.ko 2024-09-13T16:34:43.305 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mct_u232.ko 2024-09-13T16:34:43.305 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7720.ko 2024-09-13T16:34:43.305 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7840.ko 2024-09-13T16:34:43.305 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/navman.ko 2024-09-13T16:34:43.305 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/omninet.ko 2024-09-13T16:34:43.305 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/opticon.ko 2024-09-13T16:34:43.305 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/option.ko 2024-09-13T16:34:43.305 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/oti6858.ko 2024-09-13T16:34:43.306 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/pl2303.ko 2024-09-13T16:34:43.306 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcaux.ko 2024-09-13T16:34:43.306 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcserial.ko 2024-09-13T16:34:43.306 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/quatech2.ko 2024-09-13T16:34:43.306 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/safe_serial.ko 2024-09-13T16:34:43.306 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/sierra.ko 2024-09-13T16:34:43.306 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/spcp8x5.ko 2024-09-13T16:34:43.306 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ssu100.ko 2024-09-13T16:34:43.306 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/symbolserial.ko 2024-09-13T16:34:43.306 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-09-13T16:34:43.306 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-09-13T16:34:43.306 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_debug.ko 2024-09-13T16:34:43.306 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_wwan.ko 2024-09-13T16:34:43.307 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/visor.ko 2024-09-13T16:34:43.307 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/whiteheat.ko 2024-09-13T16:34:43.307 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/xsens_mt.ko 2024-09-13T16:34:43.307 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage 2024-09-13T16:34:43.307 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/uas.ko 2024-09-13T16:34:43.307 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-alauda.ko 2024-09-13T16:34:43.307 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-cypress.ko 2024-09-13T16:34:43.307 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-datafab.ko 2024-09-13T16:34:43.307 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-09-13T16:34:43.307 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-freecom.ko 2024-09-13T16:34:43.307 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-isd200.ko 2024-09-13T16:34:43.307 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-09-13T16:34:43.307 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-karma.ko 2024-09-13T16:34:43.307 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-onetouch.ko 2024-09-13T16:34:43.307 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-realtek.ko 2024-09-13T16:34:43.308 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr09.ko 2024-09-13T16:34:43.308 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr55.ko 2024-09-13T16:34:43.308 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-usbat.ko 2024-09-13T16:34:43.308 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/usb-storage.ko 2024-09-13T16:34:43.308 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip 2024-09-13T16:34:43.308 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-core.ko 2024-09-13T16:34:43.308 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-host.ko 2024-09-13T16:34:43.308 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-09-13T16:34:43.308 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio 2024-09-13T16:34:43.308 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci 2024-09-13T16:34:43.308 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-09-13T16:34:43.308 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci.ko 2024-09-13T16:34:43.308 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio.ko 2024-09-13T16:34:43.309 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-09-13T16:34:43.309 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost 2024-09-13T16:34:43.309 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost.ko 2024-09-13T16:34:43.309 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_iotlb.ko 2024-09-13T16:34:43.309 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_net.ko 2024-09-13T16:34:43.309 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_scsi.ko 2024-09-13T16:34:43.309 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video 2024-09-13T16:34:43.309 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight 2024-09-13T16:34:43.309 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/apple_bl.ko 2024-09-13T16:34:43.309 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/lcd.ko 2024-09-13T16:34:43.309 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/platform_lcd.ko 2024-09-13T16:34:43.309 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio 2024-09-13T16:34:43.309 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio.ko 2024-09-13T16:34:43.310 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_balloon.ko 2024-09-13T16:34:43.310 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_mmio.ko 2024-09-13T16:34:43.310 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci.ko 2024-09-13T16:34:43.310 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-09-13T16:34:43.310 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-09-13T16:34:43.310 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_ring.ko 2024-09-13T16:34:43.310 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1 2024-09-13T16:34:43.310 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters 2024-09-13T16:34:43.310 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2482.ko 2024-09-13T16:34:43.310 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2490.ko 2024-09-13T16:34:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves 2024-09-13T16:34:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-09-13T16:34:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-09-13T16:34:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-09-13T16:34:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-09-13T16:34:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-09-13T16:34:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-09-13T16:34:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-09-13T16:34:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-09-13T16:34:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-09-13T16:34:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_smem.ko 2024-09-13T16:34:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_therm.ko 2024-09-13T16:34:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/wire.ko 2024-09-13T16:34:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog 2024-09-13T16:34:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim1535_wdt.ko 2024-09-13T16:34:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim7101_wdt.ko 2024-09-13T16:34:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/f71808e_wdt.ko 2024-09-13T16:34:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/hpwdt.ko 2024-09-13T16:34:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/i6300esb.ko 2024-09-13T16:34:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-09-13T16:34:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_wdt.ko 2024-09-13T16:34:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ib700wdt.ko 2024-09-13T16:34:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ibmasr.ko 2024-09-13T16:34:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-09-13T16:34:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it8712f_wdt.ko 2024-09-13T16:34:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it87_wdt.ko 2024-09-13T16:34:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/machzwd.ko 2024-09-13T16:34:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/nv_tco.ko 2024-09-13T16:34:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_pci.ko 2024-09-13T16:34:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_usb.ko 2024-09-13T16:34:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-09-13T16:34:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sch311x_wdt.ko 2024-09-13T16:34:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/softdog.ko 2024-09-13T16:34:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sp5100_tco.ko 2024-09-13T16:34:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/via_wdt.ko 2024-09-13T16:34:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-09-13T16:34:43.314 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83877f_wdt.ko 2024-09-13T16:34:43.314 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83977f_wdt.ko 2024-09-13T16:34:43.314 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/wdt_pci.ko 2024-09-13T16:34:43.314 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/xen_wdt.ko 2024-09-13T16:34:43.314 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen 2024-09-13T16:34:43.314 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-acpi-processor.ko 2024-09-13T16:34:43.314 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-evtchn.ko 2024-09-13T16:34:43.314 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntalloc.ko 2024-09-13T16:34:43.314 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntdev.ko 2024-09-13T16:34:43.314 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback 2024-09-13T16:34:43.314 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-09-13T16:34:43.314 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-privcmd.ko 2024-09-13T16:34:43.315 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs 2024-09-13T16:34:43.315 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs/xenfs.ko 2024-09-13T16:34:43.315 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs 2024-09-13T16:34:43.315 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p 2024-09-13T16:34:43.315 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p/9p.ko 2024-09-13T16:34:43.315 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs 2024-09-13T16:34:43.315 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs/affs.ko 2024-09-13T16:34:43.315 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs 2024-09-13T16:34:43.315 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs/befs.ko 2024-09-13T16:34:43.315 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/binfmt_misc.ko 2024-09-13T16:34:43.315 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs 2024-09-13T16:34:43.315 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs/btrfs.ko 2024-09-13T16:34:43.316 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles 2024-09-13T16:34:43.316 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles/cachefiles.ko 2024-09-13T16:34:43.316 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph 2024-09-13T16:34:43.316 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph/ceph.ko 2024-09-13T16:34:43.316 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda 2024-09-13T16:34:43.316 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda/coda.ko 2024-09-13T16:34:43.316 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs 2024-09-13T16:34:43.316 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs/cramfs.ko 2024-09-13T16:34:43.316 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm 2024-09-13T16:34:43.317 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm/dlm.ko 2024-09-13T16:34:43.317 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs 2024-09-13T16:34:43.317 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs/ecryptfs.ko 2024-09-13T16:34:43.317 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs 2024-09-13T16:34:43.317 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs/f2fs.ko 2024-09-13T16:34:43.317 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat 2024-09-13T16:34:43.317 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/fat.ko 2024-09-13T16:34:43.317 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/msdos.ko 2024-09-13T16:34:43.317 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/vfat.ko 2024-09-13T16:34:43.317 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse 2024-09-13T16:34:43.317 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/cuse.ko 2024-09-13T16:34:43.317 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/fuse.ko 2024-09-13T16:34:43.317 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2 2024-09-13T16:34:43.317 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2/gfs2.ko 2024-09-13T16:34:43.317 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs 2024-09-13T16:34:43.318 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs/hfs.ko 2024-09-13T16:34:43.318 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus 2024-09-13T16:34:43.318 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus/hfsplus.ko 2024-09-13T16:34:43.318 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs 2024-09-13T16:34:43.318 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs/isofs.ko 2024-09-13T16:34:43.318 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs 2024-09-13T16:34:43.318 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs/jfs.ko 2024-09-13T16:34:43.318 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd 2024-09-13T16:34:43.318 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd/lockd.ko 2024-09-13T16:34:43.318 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix 2024-09-13T16:34:43.318 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix/minix.ko 2024-09-13T16:34:43.318 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs 2024-09-13T16:34:43.319 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs/netfs.ko 2024-09-13T16:34:43.319 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs 2024-09-13T16:34:43.319 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout 2024-09-13T16:34:43.319 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-09-13T16:34:43.319 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout 2024-09-13T16:34:43.319 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-09-13T16:34:43.319 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout 2024-09-13T16:34:43.319 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-09-13T16:34:43.319 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfs.ko 2024-09-13T16:34:43.319 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv3.ko 2024-09-13T16:34:43.319 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv4.ko 2024-09-13T16:34:43.319 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common 2024-09-13T16:34:43.320 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/grace.ko 2024-09-13T16:34:43.320 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/nfs_acl.ko 2024-09-13T16:34:43.320 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd 2024-09-13T16:34:43.320 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd/nfsd.ko 2024-09-13T16:34:43.320 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2 2024-09-13T16:34:43.320 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2/nilfs2.ko 2024-09-13T16:34:43.320 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls 2024-09-13T16:34:43.320 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-celtic.ko 2024-09-13T16:34:43.320 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-centeuro.ko 2024-09-13T16:34:43.320 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-croatian.ko 2024-09-13T16:34:43.320 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-cyrillic.ko 2024-09-13T16:34:43.320 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-gaelic.ko 2024-09-13T16:34:43.320 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-greek.ko 2024-09-13T16:34:43.320 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-iceland.ko 2024-09-13T16:34:43.321 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-inuit.ko 2024-09-13T16:34:43.321 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-roman.ko 2024-09-13T16:34:43.321 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-romanian.ko 2024-09-13T16:34:43.321 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-turkish.ko 2024-09-13T16:34:43.321 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1250.ko 2024-09-13T16:34:43.321 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1251.ko 2024-09-13T16:34:43.321 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1255.ko 2024-09-13T16:34:43.321 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp737.ko 2024-09-13T16:34:43.321 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp775.ko 2024-09-13T16:34:43.321 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp850.ko 2024-09-13T16:34:43.322 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp852.ko 2024-09-13T16:34:43.322 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp855.ko 2024-09-13T16:34:43.322 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp857.ko 2024-09-13T16:34:43.322 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp860.ko 2024-09-13T16:34:43.322 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp861.ko 2024-09-13T16:34:43.322 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp862.ko 2024-09-13T16:34:43.322 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp863.ko 2024-09-13T16:34:43.322 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp864.ko 2024-09-13T16:34:43.322 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp865.ko 2024-09-13T16:34:43.322 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp866.ko 2024-09-13T16:34:43.322 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp869.ko 2024-09-13T16:34:43.322 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp874.ko 2024-09-13T16:34:43.322 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp932.ko 2024-09-13T16:34:43.322 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp936.ko 2024-09-13T16:34:43.323 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp949.ko 2024-09-13T16:34:43.323 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp950.ko 2024-09-13T16:34:43.323 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_euc-jp.ko 2024-09-13T16:34:43.323 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-1.ko 2024-09-13T16:34:43.323 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-13.ko 2024-09-13T16:34:43.323 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-14.ko 2024-09-13T16:34:43.323 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-15.ko 2024-09-13T16:34:43.323 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-2.ko 2024-09-13T16:34:43.323 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-3.ko 2024-09-13T16:34:43.323 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-4.ko 2024-09-13T16:34:43.323 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-5.ko 2024-09-13T16:34:43.323 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-6.ko 2024-09-13T16:34:43.324 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-7.ko 2024-09-13T16:34:43.324 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-9.ko 2024-09-13T16:34:43.324 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-r.ko 2024-09-13T16:34:43.324 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-ru.ko 2024-09-13T16:34:43.324 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-u.ko 2024-09-13T16:34:43.324 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_ucs2_utils.ko 2024-09-13T16:34:43.324 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_utf8.ko 2024-09-13T16:34:43.324 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2 2024-09-13T16:34:43.324 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster 2024-09-13T16:34:43.324 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-09-13T16:34:43.324 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm 2024-09-13T16:34:43.324 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-09-13T16:34:43.324 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs 2024-09-13T16:34:43.325 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-09-13T16:34:43.325 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2.ko 2024-09-13T16:34:43.325 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-09-13T16:34:43.325 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-09-13T16:34:43.325 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-09-13T16:34:43.325 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs 2024-09-13T16:34:43.325 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs/overlay.ko 2024-09-13T16:34:43.325 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore 2024-09-13T16:34:43.325 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore/ramoops.ko 2024-09-13T16:34:43.325 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs 2024-09-13T16:34:43.325 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs/reiserfs.ko 2024-09-13T16:34:43.325 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs 2024-09-13T16:34:43.326 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs/romfs.ko 2024-09-13T16:34:43.326 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb 2024-09-13T16:34:43.326 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client 2024-09-13T16:34:43.326 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client/cifs.ko 2024-09-13T16:34:43.326 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common 2024-09-13T16:34:43.326 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_arc4.ko 2024-09-13T16:34:43.326 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_md4.ko 2024-09-13T16:34:43.326 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs 2024-09-13T16:34:43.326 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs/squashfs.ko 2024-09-13T16:34:43.326 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv 2024-09-13T16:34:43.326 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv/sysv.ko 2024-09-13T16:34:43.326 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf 2024-09-13T16:34:43.326 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf/udf.ko 2024-09-13T16:34:43.326 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs 2024-09-13T16:34:43.327 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs/ufs.ko 2024-09-13T16:34:43.327 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs 2024-09-13T16:34:43.327 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs/xfs.ko 2024-09-13T16:34:43.327 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel 2024-09-13T16:34:43.327 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu 2024-09-13T16:34:43.327 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu/rcutorture.ko 2024-09-13T16:34:43.327 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/torture.ko 2024-09-13T16:34:43.327 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace 2024-09-13T16:34:43.327 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-09-13T16:34:43.327 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib 2024-09-13T16:34:43.327 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/asn1_encoder.ko 2024-09-13T16:34:43.327 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc-itu-t.ko 2024-09-13T16:34:43.327 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc8.ko 2024-09-13T16:34:43.327 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto 2024-09-13T16:34:43.328 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libaescfb.ko 2024-09-13T16:34:43.328 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libarc4.ko 2024-09-13T16:34:43.328 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libdes.ko 2024-09-13T16:34:43.328 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/libcrc32c.ko 2024-09-13T16:34:43.328 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lru_cache.ko 2024-09-13T16:34:43.328 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4 2024-09-13T16:34:43.328 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4_compress.ko 2024-09-13T16:34:43.328 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4hc_compress.ko 2024-09-13T16:34:43.328 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math 2024-09-13T16:34:43.328 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math/cordic.ko 2024-09-13T16:34:43.328 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6 2024-09-13T16:34:43.328 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6/raid6_pq.ko 2024-09-13T16:34:43.328 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon 2024-09-13T16:34:43.328 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon/reed_solomon.ko 2024-09-13T16:34:43.329 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_bm.ko 2024-09-13T16:34:43.329 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_fsm.ko 2024-09-13T16:34:43.329 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_kmp.ko 2024-09-13T16:34:43.329 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd 2024-09-13T16:34:43.329 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd/zstd_compress.ko 2024-09-13T16:34:43.329 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm 2024-09-13T16:34:43.329 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm/hwpoison-inject.ko 2024-09-13T16:34:43.329 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net 2024-09-13T16:34:43.329 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan 2024-09-13T16:34:43.329 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/6lowpan.ko 2024-09-13T16:34:43.329 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_dest.ko 2024-09-13T16:34:43.329 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_fragment.ko 2024-09-13T16:34:43.329 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_hop.ko 2024-09-13T16:34:43.329 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_ipv6.ko 2024-09-13T16:34:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_mobility.ko 2024-09-13T16:34:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_routing.ko 2024-09-13T16:34:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_udp.ko 2024-09-13T16:34:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802 2024-09-13T16:34:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/garp.ko 2024-09-13T16:34:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/mrp.ko 2024-09-13T16:34:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/p8022.ko 2024-09-13T16:34:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/psnap.ko 2024-09-13T16:34:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/stp.ko 2024-09-13T16:34:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q 2024-09-13T16:34:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q/8021q.ko 2024-09-13T16:34:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p 2024-09-13T16:34:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet.ko 2024-09-13T16:34:43.330 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_fd.ko 2024-09-13T16:34:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_rdma.ko 2024-09-13T16:34:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_virtio.ko 2024-09-13T16:34:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv 2024-09-13T16:34:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv/batman-adv.ko 2024-09-13T16:34:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge 2024-09-13T16:34:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/br_netfilter.ko 2024-09-13T16:34:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/bridge.ko 2024-09-13T16:34:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter 2024-09-13T16:34:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-09-13T16:34:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_among.ko 2024-09-13T16:34:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arp.ko 2024-09-13T16:34:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-09-13T16:34:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-09-13T16:34:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip.ko 2024-09-13T16:34:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-09-13T16:34:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_limit.ko 2024-09-13T16:34:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_log.ko 2024-09-13T16:34:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark.ko 2024-09-13T16:34:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-09-13T16:34:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-09-13T16:34:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-09-13T16:34:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-09-13T16:34:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_snat.ko 2024-09-13T16:34:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_stp.ko 2024-09-13T16:34:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-09-13T16:34:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-09-13T16:34:43.332 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-09-13T16:34:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-09-13T16:34:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtables.ko 2024-09-13T16:34:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-09-13T16:34:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-09-13T16:34:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-09-13T16:34:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph 2024-09-13T16:34:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph/libceph.ko 2024-09-13T16:34:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core 2024-09-13T16:34:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/failover.ko 2024-09-13T16:34:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/pktgen.ko 2024-09-13T16:34:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp 2024-09-13T16:34:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp.ko 2024-09-13T16:34:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_diag.ko 2024-09-13T16:34:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv4.ko 2024-09-13T16:34:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv6.ko 2024-09-13T16:34:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver 2024-09-13T16:34:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver/dns_resolver.ko 2024-09-13T16:34:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa 2024-09-13T16:34:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/dsa_core.ko 2024-09-13T16:34:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_brcm.ko 2024-09-13T16:34:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_dsa.ko 2024-09-13T16:34:43.334 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_none.ko 2024-09-13T16:34:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_trailer.ko 2024-09-13T16:34:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154 2024-09-13T16:34:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan 2024-09-13T16:34:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-09-13T16:34:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154.ko 2024-09-13T16:34:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154_socket.ko 2024-09-13T16:34:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4 2024-09-13T16:34:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ah4.ko 2024-09-13T16:34:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/esp4.ko 2024-09-13T16:34:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/fou.ko 2024-09-13T16:34:43.335 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/gre.ko 2024-09-13T16:34:43.336 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/inet_diag.ko 2024-09-13T16:34:43.336 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_gre.ko 2024-09-13T16:34:43.336 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_tunnel.ko 2024-09-13T16:34:43.336 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_vti.ko 2024-09-13T16:34:43.336 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipcomp.ko 2024-09-13T16:34:43.336 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipip.ko 2024-09-13T16:34:43.336 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter 2024-09-13T16:34:43.336 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arp_tables.ko 2024-09-13T16:34:43.336 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-09-13T16:34:43.336 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-09-13T16:34:43.336 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-09-13T16:34:43.336 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-09-13T16:34:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-09-13T16:34:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-09-13T16:34:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-09-13T16:34:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-09-13T16:34:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-09-13T16:34:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_security.ko 2024-09-13T16:34:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-09-13T16:34:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-09-13T16:34:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-09-13T16:34:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-09-13T16:34:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-09-13T16:34:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-09-13T16:34:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-09-13T16:34:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-09-13T16:34:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-09-13T16:34:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-09-13T16:34:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_bic.ko 2024-09-13T16:34:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_dctcp.ko 2024-09-13T16:34:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_diag.ko 2024-09-13T16:34:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_highspeed.ko 2024-09-13T16:34:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_htcp.ko 2024-09-13T16:34:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_hybla.ko 2024-09-13T16:34:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_illinois.ko 2024-09-13T16:34:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_lp.ko 2024-09-13T16:34:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_scalable.ko 2024-09-13T16:34:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_vegas.ko 2024-09-13T16:34:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_veno.ko 2024-09-13T16:34:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_westwood.ko 2024-09-13T16:34:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_yeah.ko 2024-09-13T16:34:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tunnel4.ko 2024-09-13T16:34:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_diag.ko 2024-09-13T16:34:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_tunnel.ko 2024-09-13T16:34:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/xfrm4_tunnel.ko 2024-09-13T16:34:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6 2024-09-13T16:34:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ah6.ko 2024-09-13T16:34:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/esp6.ko 2024-09-13T16:34:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/fou6.ko 2024-09-13T16:34:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_tunnel.ko 2024-09-13T16:34:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-09-13T16:34:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_vti.ko 2024-09-13T16:34:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ipcomp6.ko 2024-09-13T16:34:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter 2024-09-13T16:34:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-09-13T16:34:43.340 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-09-13T16:34:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-09-13T16:34:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-09-13T16:34:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-09-13T16:34:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-09-13T16:34:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-09-13T16:34:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-09-13T16:34:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-09-13T16:34:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-09-13T16:34:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-09-13T16:34:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-09-13T16:34:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-09-13T16:34:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-09-13T16:34:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-09-13T16:34:43.342 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-09-13T16:34:43.342 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-09-13T16:34:43.342 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-09-13T16:34:43.342 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-09-13T16:34:43.342 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-09-13T16:34:43.342 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-09-13T16:34:43.342 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-09-13T16:34:43.342 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-09-13T16:34:43.342 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-09-13T16:34:43.342 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/sit.ko 2024-09-13T16:34:43.342 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/tunnel6.ko 2024-09-13T16:34:43.342 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/xfrm6_tunnel.ko 2024-09-13T16:34:43.342 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key 2024-09-13T16:34:43.343 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key/af_key.ko 2024-09-13T16:34:43.343 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp 2024-09-13T16:34:43.343 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_core.ko 2024-09-13T16:34:43.343 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_debugfs.ko 2024-09-13T16:34:43.343 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_eth.ko 2024-09-13T16:34:43.343 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip.ko 2024-09-13T16:34:43.343 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip6.ko 2024-09-13T16:34:43.343 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_netlink.ko 2024-09-13T16:34:43.343 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ppp.ko 2024-09-13T16:34:43.343 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc 2024-09-13T16:34:43.343 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc/llc.ko 2024-09-13T16:34:43.343 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154 2024-09-13T16:34:43.343 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154/mac802154.ko 2024-09-13T16:34:43.344 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls 2024-09-13T16:34:43.344 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls/mpls_gso.ko 2024-09-13T16:34:43.344 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter 2024-09-13T16:34:43.344 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset 2024-09-13T16:34:43.344 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set.ko 2024-09-13T16:34:43.344 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-09-13T16:34:43.344 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-09-13T16:34:43.344 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-09-13T16:34:43.344 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-09-13T16:34:43.344 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-09-13T16:34:43.344 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-09-13T16:34:43.344 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-09-13T16:34:43.344 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-09-13T16:34:43.345 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-09-13T16:34:43.345 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-09-13T16:34:43.345 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-09-13T16:34:43.345 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-09-13T16:34:43.345 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-09-13T16:34:43.345 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-09-13T16:34:43.345 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-09-13T16:34:43.345 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs 2024-09-13T16:34:43.345 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs.ko 2024-09-13T16:34:43.345 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-09-13T16:34:43.345 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-09-13T16:34:43.345 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-09-13T16:34:43.345 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-09-13T16:34:43.345 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-09-13T16:34:43.346 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-09-13T16:34:43.346 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-09-13T16:34:43.346 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-09-13T16:34:43.346 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-09-13T16:34:43.346 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-09-13T16:34:43.346 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-09-13T16:34:43.346 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-09-13T16:34:43.346 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-09-13T16:34:43.346 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conncount.ko 2024-09-13T16:34:43.346 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack.ko 2024-09-13T16:34:43.346 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-09-13T16:34:43.346 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-09-13T16:34:43.346 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-09-13T16:34:43.347 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_h323.ko 2024-09-13T16:34:43.347 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_irc.ko 2024-09-13T16:34:43.347 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-09-13T16:34:43.347 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-09-13T16:34:43.347 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-09-13T16:34:43.347 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sane.ko 2024-09-13T16:34:43.347 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sip.ko 2024-09-13T16:34:43.347 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-09-13T16:34:43.347 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-09-13T16:34:43.347 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_dup_netdev.ko 2024-09-13T16:34:43.347 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table.ko 2024-09-13T16:34:43.348 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table_inet.ko 2024-09-13T16:34:43.348 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_log_syslog.ko 2024-09-13T16:34:43.348 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat.ko 2024-09-13T16:34:43.348 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_amanda.ko 2024-09-13T16:34:43.348 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_ftp.ko 2024-09-13T16:34:43.348 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_irc.ko 2024-09-13T16:34:43.348 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_sip.ko 2024-09-13T16:34:43.348 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_tftp.ko 2024-09-13T16:34:43.348 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_synproxy_core.ko 2024-09-13T16:34:43.348 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_tables.ko 2024-09-13T16:34:43.348 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink.ko 2024-09-13T16:34:43.349 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_acct.ko 2024-09-13T16:34:43.349 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_log.ko 2024-09-13T16:34:43.349 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_osf.ko 2024-09-13T16:34:43.349 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_queue.ko 2024-09-13T16:34:43.349 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_chain_nat.ko 2024-09-13T16:34:43.349 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_compat.ko 2024-09-13T16:34:43.349 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_ct.ko 2024-09-13T16:34:43.349 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_dup_netdev.ko 2024-09-13T16:34:43.349 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fib.ko 2024-09-13T16:34:43.349 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fwd_netdev.ko 2024-09-13T16:34:43.349 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_hash.ko 2024-09-13T16:34:43.350 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_limit.ko 2024-09-13T16:34:43.350 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_log.ko 2024-09-13T16:34:43.350 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_masq.ko 2024-09-13T16:34:43.350 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_nat.ko 2024-09-13T16:34:43.350 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_numgen.ko 2024-09-13T16:34:43.350 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_queue.ko 2024-09-13T16:34:43.350 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_quota.ko 2024-09-13T16:34:43.350 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_redir.ko 2024-09-13T16:34:43.351 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject.ko 2024-09-13T16:34:43.351 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject_inet.ko 2024-09-13T16:34:43.351 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_socket.ko 2024-09-13T16:34:43.351 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_tunnel.ko 2024-09-13T16:34:43.351 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_AUDIT.ko 2024-09-13T16:34:43.351 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CHECKSUM.ko 2024-09-13T16:34:43.351 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CLASSIFY.ko 2024-09-13T16:34:43.351 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-09-13T16:34:43.351 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CT.ko 2024-09-13T16:34:43.351 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_DSCP.ko 2024-09-13T16:34:43.351 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HL.ko 2024-09-13T16:34:43.351 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HMARK.ko 2024-09-13T16:34:43.351 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_IDLETIMER.ko 2024-09-13T16:34:43.352 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LED.ko 2024-09-13T16:34:43.352 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LOG.ko 2024-09-13T16:34:43.352 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_MASQUERADE.ko 2024-09-13T16:34:43.352 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NETMAP.ko 2024-09-13T16:34:43.352 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFLOG.ko 2024-09-13T16:34:43.352 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFQUEUE.ko 2024-09-13T16:34:43.352 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_RATEEST.ko 2024-09-13T16:34:43.352 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_REDIRECT.ko 2024-09-13T16:34:43.352 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_SECMARK.ko 2024-09-13T16:34:43.352 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPMSS.ko 2024-09-13T16:34:43.352 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-09-13T16:34:43.352 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TEE.ko 2024-09-13T16:34:43.353 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TPROXY.ko 2024-09-13T16:34:43.353 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TRACE.ko 2024-09-13T16:34:43.353 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_addrtype.ko 2024-09-13T16:34:43.353 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_bpf.ko 2024-09-13T16:34:43.353 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cgroup.ko 2024-09-13T16:34:43.353 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cluster.ko 2024-09-13T16:34:43.353 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_comment.ko 2024-09-13T16:34:43.353 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connbytes.ko 2024-09-13T16:34:43.353 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlabel.ko 2024-09-13T16:34:43.353 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlimit.ko 2024-09-13T16:34:43.353 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connmark.ko 2024-09-13T16:34:43.353 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_conntrack.ko 2024-09-13T16:34:43.353 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cpu.ko 2024-09-13T16:34:43.353 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dccp.ko 2024-09-13T16:34:43.354 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_devgroup.ko 2024-09-13T16:34:43.354 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dscp.ko 2024-09-13T16:34:43.354 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ecn.ko 2024-09-13T16:34:43.354 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_esp.ko 2024-09-13T16:34:43.354 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hashlimit.ko 2024-09-13T16:34:43.354 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_helper.ko 2024-09-13T16:34:43.354 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hl.ko 2024-09-13T16:34:43.354 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipcomp.ko 2024-09-13T16:34:43.354 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_iprange.ko 2024-09-13T16:34:43.354 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipvs.ko 2024-09-13T16:34:43.354 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_l2tp.ko 2024-09-13T16:34:43.354 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_length.ko 2024-09-13T16:34:43.355 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_limit.ko 2024-09-13T16:34:43.355 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mac.ko 2024-09-13T16:34:43.355 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mark.ko 2024-09-13T16:34:43.355 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_multiport.ko 2024-09-13T16:34:43.355 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nat.ko 2024-09-13T16:34:43.355 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nfacct.ko 2024-09-13T16:34:43.355 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_osf.ko 2024-09-13T16:34:43.355 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_owner.ko 2024-09-13T16:34:43.355 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_physdev.ko 2024-09-13T16:34:43.355 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_pkttype.ko 2024-09-13T16:34:43.355 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_policy.ko 2024-09-13T16:34:43.355 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_quota.ko 2024-09-13T16:34:43.355 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_rateest.ko 2024-09-13T16:34:43.356 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_realm.ko 2024-09-13T16:34:43.356 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_recent.ko 2024-09-13T16:34:43.356 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_sctp.ko 2024-09-13T16:34:43.356 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_set.ko 2024-09-13T16:34:43.356 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_socket.ko 2024-09-13T16:34:43.356 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_state.ko 2024-09-13T16:34:43.356 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_statistic.ko 2024-09-13T16:34:43.356 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_string.ko 2024-09-13T16:34:43.356 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_tcpmss.ko 2024-09-13T16:34:43.356 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_time.ko 2024-09-13T16:34:43.356 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_u32.ko 2024-09-13T16:34:43.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink 2024-09-13T16:34:43.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink/netlink_diag.ko 2024-09-13T16:34:43.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh 2024-09-13T16:34:43.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh/nsh.ko 2024-09-13T16:34:43.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch 2024-09-13T16:34:43.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/openvswitch.ko 2024-09-13T16:34:43.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-geneve.ko 2024-09-13T16:34:43.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-gre.ko 2024-09-13T16:34:43.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-vxlan.ko 2024-09-13T16:34:43.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet 2024-09-13T16:34:43.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet/af_packet_diag.ko 2024-09-13T16:34:43.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds 2024-09-13T16:34:43.357 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds.ko 2024-09-13T16:34:43.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_rdma.ko 2024-09-13T16:34:43.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_tcp.ko 2024-09-13T16:34:43.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched 2024-09-13T16:34:43.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_csum.ko 2024-09-13T16:34:43.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_gact.ko 2024-09-13T16:34:43.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_mirred.ko 2024-09-13T16:34:43.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_nat.ko 2024-09-13T16:34:43.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_pedit.ko 2024-09-13T16:34:43.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_police.ko 2024-09-13T16:34:43.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_simple.ko 2024-09-13T16:34:43.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_skbedit.ko 2024-09-13T16:34:43.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_vlan.ko 2024-09-13T16:34:43.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_basic.ko 2024-09-13T16:34:43.358 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_bpf.ko 2024-09-13T16:34:43.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_flow.ko 2024-09-13T16:34:43.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_fw.ko 2024-09-13T16:34:43.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_route.ko 2024-09-13T16:34:43.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_u32.ko 2024-09-13T16:34:43.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_cmp.ko 2024-09-13T16:34:43.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_ipset.ko 2024-09-13T16:34:43.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_meta.ko 2024-09-13T16:34:43.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_nbyte.ko 2024-09-13T16:34:43.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_text.ko 2024-09-13T16:34:43.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_u32.ko 2024-09-13T16:34:43.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_choke.ko 2024-09-13T16:34:43.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_codel.ko 2024-09-13T16:34:43.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_drr.ko 2024-09-13T16:34:43.359 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_fq.ko 2024-09-13T16:34:43.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_gred.ko 2024-09-13T16:34:43.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hfsc.ko 2024-09-13T16:34:43.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hhf.ko 2024-09-13T16:34:43.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_htb.ko 2024-09-13T16:34:43.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_ingress.ko 2024-09-13T16:34:43.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio.ko 2024-09-13T16:34:43.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio_lib.ko 2024-09-13T16:34:43.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_multiq.ko 2024-09-13T16:34:43.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_netem.ko 2024-09-13T16:34:43.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_pie.ko 2024-09-13T16:34:43.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_plug.ko 2024-09-13T16:34:43.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_prio.ko 2024-09-13T16:34:43.360 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_qfq.ko 2024-09-13T16:34:43.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_red.ko 2024-09-13T16:34:43.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfb.ko 2024-09-13T16:34:43.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfq.ko 2024-09-13T16:34:43.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_tbf.ko 2024-09-13T16:34:43.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_teql.ko 2024-09-13T16:34:43.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp 2024-09-13T16:34:43.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp.ko 2024-09-13T16:34:43.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp_diag.ko 2024-09-13T16:34:43.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc 2024-09-13T16:34:43.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss 2024-09-13T16:34:43.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-09-13T16:34:43.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-09-13T16:34:43.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/sunrpc.ko 2024-09-13T16:34:43.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma 2024-09-13T16:34:43.361 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-09-13T16:34:43.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc 2024-09-13T16:34:43.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc.ko 2024-09-13T16:34:43.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc_diag.ko 2024-09-13T16:34:43.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix 2024-09-13T16:34:43.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix/unix_diag.ko 2024-09-13T16:34:43.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock 2024-09-13T16:34:43.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-09-13T16:34:43.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-09-13T16:34:43.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock.ko 2024-09-13T16:34:43.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_diag.ko 2024-09-13T16:34:43.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_loopback.ko 2024-09-13T16:34:43.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm 2024-09-13T16:34:43.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm/xfrm_ipcomp.ko 2024-09-13T16:34:43.362 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security 2024-09-13T16:34:43.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys 2024-09-13T16:34:43.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys 2024-09-13T16:34:43.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-09-13T16:34:43.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys 2024-09-13T16:34:43.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys/trusted.ko 2024-09-13T16:34:43.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias 2024-09-13T16:34:43.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias.bin 2024-09-13T16:34:43.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin 2024-09-13T16:34:43.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.alias.bin 2024-09-13T16:34:43.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.bin 2024-09-13T16:34:43.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.modinfo 2024-09-13T16:34:43.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep 2024-09-13T16:34:43.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep.bin 2024-09-13T16:34:43.363 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.devname 2024-09-13T16:34:43.364 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.order 2024-09-13T16:34:43.364 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.softdep 2024-09-13T16:34:43.364 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols 2024-09-13T16:34:43.364 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols.bin 2024-09-13T16:34:43.364 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.weakdep 2024-09-13T16:34:43.364 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-13T16:34:43.368 DEBUG:teuthology.task.kernel:get_image_version: 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:34:43.368 DEBUG:teuthology.orchestra.run.smithi028:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-09-13T16:34:43.392 INFO:teuthology.orchestra.run.smithi028.stdout:postinstall scriptlet (using /bin/sh): 2024-09-13T16:34:43.392 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-13T16:34:43.392 INFO:teuthology.orchestra.run.smithi028.stdout: /usr/bin/kernel-install add 6.11.0-rc5-g2ca44a2f35ca /lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-13T16:34:43.392 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-09-13T16:34:43.393 INFO:teuthology.orchestra.run.smithi028.stdout:for file in vmlinuz System.map config; do 2024-09-13T16:34:43.393 INFO:teuthology.orchestra.run.smithi028.stdout: if ! cmp --silent "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca"; then 2024-09-13T16:34:43.393 INFO:teuthology.orchestra.run.smithi028.stdout: cp "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca" 2024-09-13T16:34:43.393 INFO:teuthology.orchestra.run.smithi028.stdout: fi 2024-09-13T16:34:43.393 INFO:teuthology.orchestra.run.smithi028.stdout:done 2024-09-13T16:34:43.393 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -d "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" ] && \ 2024-09-13T16:34:43.393 INFO:teuthology.orchestra.run.smithi028.stdout: ! diff -rq "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" >/dev/null 2>&1; then 2024-09-13T16:34:43.393 INFO:teuthology.orchestra.run.smithi028.stdout: rm -rf "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-13T16:34:43.393 INFO:teuthology.orchestra.run.smithi028.stdout: cp -r "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-13T16:34:43.393 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-09-13T16:34:43.393 INFO:teuthology.orchestra.run.smithi028.stdout:if [ ! -e "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep" ]; then 2024-09-13T16:34:43.393 INFO:teuthology.orchestra.run.smithi028.stdout: /usr/sbin/depmod 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:34:43.393 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-09-13T16:34:43.394 INFO:teuthology.orchestra.run.smithi028.stdout:preuninstall scriptlet (using /bin/sh): 2024-09-13T16:34:43.394 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-13T16:34:43.394 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-install remove 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:34:43.394 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-09-13T16:34:43.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-09-13T16:34:43.472 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-09-13T16:34:43.473 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-09-13T16:34:43.473 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-09-13T16:34:43.473 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 82.el9 2024-09-13T16:34:43.473 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-09-13T16:34:43.473 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T16:34:43.473 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-09-13T16:34:43.473 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8274670 2024-09-13T16:34:43.473 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-09-13T16:34:43.473 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T16:34:43.473 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T16:34:43.474 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T16:34:43.474 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T16:34:43.474 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-09-13T16:34:43.474 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-09-13T16:34:43.474 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T16:34:43.474 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-09-13T16:34:43.474 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-09-13T16:34:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:34:43.474 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T16:34:43.474 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T16:34:43.474 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T16:34:43.475 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-09-13T16:34:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:34:43.475 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T16:34:43.476 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T16:34:43.476 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:34:43.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T16:34:44.051 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-09-13T16:34:45.316 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T16:34:45.331 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-09-13T16:34:45.333 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-09-13T16:34:45.348 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.ChCToTqhQt 2024-09-13T16:34:45.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ChCToTqhQt 2024-09-13T16:34:45.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.ChCToTqhQt 2024-09-13T16:34:45.545 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.ChCToTqhQt is 6KB 2024-09-13T16:34:45.594 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.ChCToTqhQt 2024-09-13T16:34:45.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T16:34:45.679 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T16:34:45.679 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T16:34:45.680 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T16:34:45.680 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca.conf 2024-09-13T16:34:45.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:34:45.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-09-13T16:35:15.838 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-13T16:35:15.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-13T16:35:15.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:35:34.333 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-09-13T16:35:43.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-13T16:35:43.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:36:01.729 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-09-13T16:36:13.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-13T16:36:13.731 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:36:16.801 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-09-13T16:36:31.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-13T16:36:31.802 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:36:32.236 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-09-13T16:36:32.963 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-13T16:36:32.964 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.11.0-rc5-g2ca44a2f35ca"... 2024-09-13T16:36:32.964 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-09-13T16:36:33.040 INFO:teuthology.orchestra.run.smithi028.stdout:6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:36:33.040 DEBUG:teuthology.task.kernel:current kernel version is 6.11.0-rc5-g2ca44a2f35ca vs 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:36:33.041 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T16:36:33.041 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T16:36:34.041 DEBUG:teuthology.parallel:result is None 2024-09-13T16:36:34.042 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T16:36:34.048 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T16:36:34.049 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T16:36:34.055 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T16:36:34.086 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T16:36:34.093 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T16:36:34.100 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T16:36:34.100 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T16:36:34.130 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T16:36:34.323 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T16:36:34.330 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T16:36:34.330 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T16:36:34.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:36:34.348 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T16:36:34.351 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T16:36:34.451 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:36:34.469 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:36:34.718 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:36:34.852 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:36:34.855 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T16:36:34.862 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T16:36:34.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T16:36:34.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T16:36:34.921 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T16:36:34.930 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T16:36:34.930 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T16:36:34.934 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T16:36:34.965 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T16:36:35.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T16:36:35.149 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T16:36:35.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T16:36:35.282 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:36:35.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T16:36:35.337 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T16:36:35.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T16:36:35.439 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T16:36:35.540 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T16:36:35.568 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:36:35.568 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T16:36:35.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-09-13T16:36:35.638 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2024-09-13T16:36:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T16:36:35.723 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T16:36:36.223 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T16:36:36.233 INFO:teuthology.task.internal:Starting timer... 2024-09-13T16:36:36.233 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T16:36:36.243 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T16:36:36.251 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-13T16:36:36.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-09-13T16:36:36.356 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-09-13T16:36:36.359 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2024-09-13T16:36:36.422 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2024-09-13T16:36:36.424 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T16:36:36.424 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-09-13T16:36:36.464 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-09-13T16:36:36.465 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2024-09-13T16:36:36.492 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2024-09-13T16:36:36.492 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2024-09-13T16:36:36.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T16:36:36.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:36:36.562 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T16:36:36.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:36:36.592 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T16:36:36.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-09-13T16:36:36.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2024-09-13T16:36:36.688 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T16:36:36.697 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T16:36:36.709 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T16:36:36.710 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T16:41:22.274 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2024-09-13T16:41:22.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-13T16:41:22.276 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:41:22.376 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-09-13T16:41:22.483 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-13T16:41:22.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-13T16:41:22.484 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:41:22.566 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-09-13T16:41:22.646 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-13T16:41:22.647 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T16:41:22.657 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T16:41:22.657 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T16:41:22.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T16:41:22.662 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T16:41:22.662 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T16:41:22.719 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T16:41:22.719 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T16:41:22.735 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T16:41:22.749 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T16:41:22.759 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2024-09-13T16:41:22.771 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2024-09-13T16:41:22.785 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T16:41:22.799 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T16:41:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-09-13T16:41:22.825 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-09-13T16:41:22.855 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T16:41:22.864 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-09-13T16:41:22.867 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T16:41:22.868 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-09-13T16:41:22.868 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T16:41:22.868 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T16:41:22.868 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T16:41:22.868 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T16:41:22.884 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T16:41:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-09-13T16:41:22.982 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T16:41:22.982 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-09-13T16:41:22.982 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T16:41:22.982 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T16:41:22.982 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T16:41:22.982 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T16:41:22.983 INFO:teuthology.run_tasks:Running task install... 2024-09-13T16:41:22.991 DEBUG:teuthology.task.install:project ceph 2024-09-13T16:41:22.991 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '50bbe7f395997dcd871813949c49addbbf630361'}} 2024-09-13T16:41:22.992 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '50bbe7f395997dcd871813949c49addbbf630361'} 2024-09-13T16:41:22.992 INFO:teuthology.task.install:Using flavor: default 2024-09-13T16:41:22.997 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T16:41:22.998 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-09-13T16:41:22.998 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '50bbe7f395997dcd871813949c49addbbf630361', 'tag': None, 'wait_for_package': False} 2024-09-13T16:41:22.998 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T16:41:23.000 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '50bbe7f395997dcd871813949c49addbbf630361', 'tag': None, 'wait_for_package': False} 2024-09-13T16:41:23.001 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T16:41:23.180 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/ 2024-09-13T16:41:23.181 INFO:teuthology.task.install.rpm:Package version is 18.2.4-850.g50bbe7f3 2024-09-13T16:41:23.192 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/ 2024-09-13T16:41:23.193 INFO:teuthology.task.install.rpm:Package version is 18.2.4-850.g50bbe7f3 2024-09-13T16:41:23.301 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T16:41:23.302 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:41:23.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T16:41:23.312 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T16:41:23.312 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:41:23.312 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T16:41:23.338 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-13T16:41:23.338 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/50bbe7f395997dcd871813949c49addbbf630361/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T16:41:23.360 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-13T16:41:23.360 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/50bbe7f395997dcd871813949c49addbbf630361/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T16:41:23.403 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T16:41:23.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T16:41:23.478 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T16:41:23.537 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T16:41:23.543 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2024-09-13T16:41:23.545 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2024-09-13T16:41:23.593 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-09-13T16:41:23.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-09-13T16:41:23.770 INFO:teuthology.orchestra.run.smithi191.stdout:45 files removed 2024-09-13T16:41:23.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-13T16:41:23.872 INFO:teuthology.orchestra.run.smithi028.stdout:45 files removed 2024-09-13T16:41:23.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-13T16:41:24.357 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 276 kB/s | 77 kB 00:00 2024-09-13T16:41:24.632 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 49 kB/s | 11 kB 00:00 2024-09-13T16:41:24.665 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 237 kB/s | 77 kB 00:00 2024-09-13T16:41:24.908 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 7.8 kB/s | 1.9 kB 00:00 2024-09-13T16:41:25.016 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 38 kB/s | 11 kB 00:00 2024-09-13T16:41:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 6.7 kB/s | 1.9 kB 00:00 2024-09-13T16:41:26.800 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.2 MB 00:01 2024-09-13T16:41:27.008 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.2 MB 00:01 2024-09-13T16:41:28.743 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-13T16:41:28.934 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 28 MB/s | 20 MB 00:00 2024-09-13T16:41:33.961 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.5 MB 00:00 2024-09-13T16:41:35.468 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 6.5 MB 00:02 2024-09-13T16:41:35.630 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 19 kB 00:00 2024-09-13T16:41:36.926 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-13T16:41:37.144 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 19 kB 00:00 2024-09-13T16:41:38.331 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-13T16:41:43.557 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-09-13T16:41:43.704 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-13T16:41:46.025 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-13T16:41:46.025 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-13T16:41:46.038 INFO:teuthology.orchestra.run.smithi191.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-13T16:41:46.039 INFO:teuthology.orchestra.run.smithi191.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-13T16:41:46.040 INFO:teuthology.orchestra.run.smithi191.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-13T16:41:46.041 INFO:teuthology.orchestra.run.smithi191.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-13T16:41:46.043 INFO:teuthology.orchestra.run.smithi191.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-13T16:41:46.045 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-13T16:41:46.046 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-13T16:41:46.107 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-13T16:41:46.121 INFO:teuthology.orchestra.run.smithi191.stdout:====================================================================================== 2024-09-13T16:41:46.121 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-13T16:41:46.121 INFO:teuthology.orchestra.run.smithi191.stdout:====================================================================================== 2024-09-13T16:41:46.121 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-13T16:41:46.122 INFO:teuthology.orchestra.run.smithi191.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-13T16:41:46.122 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 6.5 k 2024-09-13T16:41:46.122 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 5.1 M 2024-09-13T16:41:46.122 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 852 k 2024-09-13T16:41:46.122 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 143 k 2024-09-13T16:41:46.122 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 1.5 M 2024-09-13T16:41:46.122 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 139 k 2024-09-13T16:41:46.122 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 3.5 M 2024-09-13T16:41:46.122 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 7.4 M 2024-09-13T16:41:46.122 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 49 k 2024-09-13T16:41:46.122 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 7.8 M 2024-09-13T16:41:46.122 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 40 M 2024-09-13T16:41:46.123 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 264 k 2024-09-13T16:41:46.123 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 226 k 2024-09-13T16:41:46.123 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 224 k 2024-09-13T16:41:46.123 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 18 k 2024-09-13T16:41:46.123 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-13T16:41:46.123 INFO:teuthology.orchestra.run.smithi191.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-13T16:41:46.123 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-13T16:41:46.123 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 31 k 2024-09-13T16:41:46.123 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 712 k 2024-09-13T16:41:46.123 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 127 k 2024-09-13T16:41:46.123 INFO:teuthology.orchestra.run.smithi191.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-13T16:41:46.123 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-13T16:41:46.123 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-13T16:41:46.124 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-13T16:41:46.124 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-13T16:41:46.124 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-13T16:41:46.124 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 162 k 2024-09-13T16:41:46.124 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 319 k 2024-09-13T16:41:46.124 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 301 k 2024-09-13T16:41:46.124 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 99 k 2024-09-13T16:41:46.124 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 87 k 2024-09-13T16:41:46.124 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 3.0 M 2024-09-13T16:41:46.124 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 172 k 2024-09-13T16:41:46.124 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-13T16:41:46.124 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2024-09-13T16:41:46.125 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-13T16:41:46.125 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-13T16:41:46.125 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-13T16:41:46.125 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-13T16:41:46.125 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 3.3 M 2024-09-13T16:41:46.125 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 3.0 M 2024-09-13T16:41:46.125 INFO:teuthology.orchestra.run.smithi191.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-13T16:41:46.125 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-13T16:41:46.125 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-13T16:41:46.125 INFO:teuthology.orchestra.run.smithi191.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-13T16:41:46.125 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T16:41:46.125 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 18 M 2024-09-13T16:41:46.125 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 24 k 2024-09-13T16:41:46.126 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 2.1 M 2024-09-13T16:41:46.126 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 246 k 2024-09-13T16:41:46.126 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 4.7 M 2024-09-13T16:41:46.126 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 17 M 2024-09-13T16:41:46.126 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 16 k 2024-09-13T16:41:46.127 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 25 k 2024-09-13T16:41:46.127 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T16:41:46.127 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T16:41:46.127 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T16:41:46.127 INFO:teuthology.orchestra.run.smithi191.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-13T16:41:46.127 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T16:41:46.127 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T16:41:46.127 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T16:41:46.127 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T16:41:46.127 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 166 k 2024-09-13T16:41:46.127 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T16:41:46.127 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T16:41:46.128 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T16:41:46.128 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T16:41:46.128 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T16:41:46.128 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 477 k 2024-09-13T16:41:46.129 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T16:41:46.129 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 4.5 M 2024-09-13T16:41:46.129 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T16:41:46.129 INFO:teuthology.orchestra.run.smithi191.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-13T16:41:46.129 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T16:41:46.129 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T16:41:46.129 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T16:41:46.129 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T16:41:46.129 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-13T16:41:46.129 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T16:41:46.130 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T16:41:46.130 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T16:41:46.130 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T16:41:46.130 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 45 k 2024-09-13T16:41:46.130 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 129 k 2024-09-13T16:41:46.130 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T16:41:46.130 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T16:41:46.130 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T16:41:46.130 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T16:41:46.130 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T16:41:46.130 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T16:41:46.130 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T16:41:46.131 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T16:41:46.131 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T16:41:46.131 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-13T16:41:46.131 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T16:41:46.131 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T16:41:46.131 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T16:41:46.131 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T16:41:46.131 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T16:41:46.131 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T16:41:46.131 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T16:41:46.131 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T16:41:46.131 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T16:41:46.131 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T16:41:46.132 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T16:41:46.132 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T16:41:46.132 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T16:41:46.132 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T16:41:46.132 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T16:41:46.132 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T16:41:46.132 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T16:41:46.132 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T16:41:46.132 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T16:41:46.132 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T16:41:46.132 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T16:41:46.132 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T16:41:46.132 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T16:41:46.132 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T16:41:46.133 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T16:41:46.133 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T16:41:46.133 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T16:41:46.133 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T16:41:46.133 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T16:41:46.133 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T16:41:46.133 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T16:41:46.133 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T16:41:46.133 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T16:41:46.133 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T16:41:46.133 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T16:41:46.133 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T16:41:46.133 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T16:41:46.134 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T16:41:46.134 INFO:teuthology.orchestra.run.smithi191.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T16:41:46.134 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T16:41:46.134 INFO:teuthology.orchestra.run.smithi191.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T16:41:46.134 INFO:teuthology.orchestra.run.smithi191.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-13T16:41:46.134 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T16:41:46.134 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-13T16:41:46.134 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:41:46.134 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-13T16:41:46.134 INFO:teuthology.orchestra.run.smithi191.stdout:====================================================================================== 2024-09-13T16:41:46.134 INFO:teuthology.orchestra.run.smithi191.stdout:Install 130 Packages 2024-09-13T16:41:46.134 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 8 Packages 2024-09-13T16:41:46.134 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:41:46.135 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 203 M 2024-09-13T16:41:46.135 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-13T16:41:46.138 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-13T16:41:46.139 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-13T16:41:46.152 INFO:teuthology.orchestra.run.smithi028.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-13T16:41:46.153 INFO:teuthology.orchestra.run.smithi028.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-13T16:41:46.154 INFO:teuthology.orchestra.run.smithi028.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-13T16:41:46.155 INFO:teuthology.orchestra.run.smithi028.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-13T16:41:46.157 INFO:teuthology.orchestra.run.smithi028.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-13T16:41:46.159 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-13T16:41:46.161 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-13T16:41:46.227 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-13T16:41:46.246 INFO:teuthology.orchestra.run.smithi028.stdout:====================================================================================== 2024-09-13T16:41:46.246 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-09-13T16:41:46.246 INFO:teuthology.orchestra.run.smithi028.stdout:====================================================================================== 2024-09-13T16:41:46.246 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-13T16:41:46.247 INFO:teuthology.orchestra.run.smithi028.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-13T16:41:46.247 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 6.5 k 2024-09-13T16:41:46.247 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 5.1 M 2024-09-13T16:41:46.247 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 852 k 2024-09-13T16:41:46.247 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 143 k 2024-09-13T16:41:46.247 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 1.5 M 2024-09-13T16:41:46.247 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 139 k 2024-09-13T16:41:46.247 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 3.5 M 2024-09-13T16:41:46.247 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 7.4 M 2024-09-13T16:41:46.247 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 49 k 2024-09-13T16:41:46.247 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 7.8 M 2024-09-13T16:41:46.247 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 40 M 2024-09-13T16:41:46.247 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 264 k 2024-09-13T16:41:46.248 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 226 k 2024-09-13T16:41:46.248 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 224 k 2024-09-13T16:41:46.248 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 18 k 2024-09-13T16:41:46.248 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-13T16:41:46.248 INFO:teuthology.orchestra.run.smithi028.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-13T16:41:46.248 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-13T16:41:46.248 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 31 k 2024-09-13T16:41:46.248 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 712 k 2024-09-13T16:41:46.248 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 127 k 2024-09-13T16:41:46.248 INFO:teuthology.orchestra.run.smithi028.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-13T16:41:46.248 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-13T16:41:46.248 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-13T16:41:46.248 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-13T16:41:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-13T16:41:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-13T16:41:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 162 k 2024-09-13T16:41:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 319 k 2024-09-13T16:41:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 301 k 2024-09-13T16:41:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 99 k 2024-09-13T16:41:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 87 k 2024-09-13T16:41:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 3.0 M 2024-09-13T16:41:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 172 k 2024-09-13T16:41:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-13T16:41:46.249 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-09-13T16:41:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-13T16:41:46.249 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-13T16:41:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-13T16:41:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-13T16:41:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 3.3 M 2024-09-13T16:41:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 3.0 M 2024-09-13T16:41:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-13T16:41:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-13T16:41:46.250 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-13T16:41:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-13T16:41:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T16:41:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 18 M 2024-09-13T16:41:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 24 k 2024-09-13T16:41:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 2.1 M 2024-09-13T16:41:46.250 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 246 k 2024-09-13T16:41:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 4.7 M 2024-09-13T16:41:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 17 M 2024-09-13T16:41:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 16 k 2024-09-13T16:41:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 25 k 2024-09-13T16:41:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T16:41:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T16:41:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T16:41:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-13T16:41:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T16:41:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T16:41:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T16:41:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T16:41:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 166 k 2024-09-13T16:41:46.251 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T16:41:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T16:41:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T16:41:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T16:41:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T16:41:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 477 k 2024-09-13T16:41:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T16:41:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 4.5 M 2024-09-13T16:41:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T16:41:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-13T16:41:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T16:41:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T16:41:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T16:41:46.252 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T16:41:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-13T16:41:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T16:41:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T16:41:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T16:41:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T16:41:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 45 k 2024-09-13T16:41:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 129 k 2024-09-13T16:41:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T16:41:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T16:41:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T16:41:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T16:41:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T16:41:46.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T16:41:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T16:41:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T16:41:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T16:41:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-13T16:41:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T16:41:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T16:41:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T16:41:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T16:41:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T16:41:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T16:41:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T16:41:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T16:41:46.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T16:41:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T16:41:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T16:41:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T16:41:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T16:41:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T16:41:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T16:41:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T16:41:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T16:41:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T16:41:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T16:41:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T16:41:46.255 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T16:41:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T16:41:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T16:41:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T16:41:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T16:41:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T16:41:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T16:41:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T16:41:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T16:41:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T16:41:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T16:41:46.256 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T16:41:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T16:41:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T16:41:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T16:41:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T16:41:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T16:41:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T16:41:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T16:41:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T16:41:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T16:41:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-13T16:41:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T16:41:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-13T16:41:46.257 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:41:46.258 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-13T16:41:46.258 INFO:teuthology.orchestra.run.smithi028.stdout:====================================================================================== 2024-09-13T16:41:46.258 INFO:teuthology.orchestra.run.smithi028.stdout:Install 130 Packages 2024-09-13T16:41:46.258 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 8 Packages 2024-09-13T16:41:46.258 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:41:46.258 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 203 M 2024-09-13T16:41:46.258 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-13T16:41:46.525 INFO:teuthology.orchestra.run.smithi028.stdout:(1/138): ceph-18.2.4-850.g50bbe7f3.el9.x86_64.r 44 kB/s | 6.5 kB 00:00 2024-09-13T16:41:46.691 INFO:teuthology.orchestra.run.smithi028.stdout:(2/138): ceph-fuse-18.2.4-850.g50bbe7f3.el9.x86 5.0 MB/s | 852 kB 00:00 2024-09-13T16:41:46.705 INFO:teuthology.orchestra.run.smithi191.stdout:(1/138): ceph-18.2.4-850.g50bbe7f3.el9.x86_64.r 48 kB/s | 6.5 kB 00:00 2024-09-13T16:41:46.758 INFO:teuthology.orchestra.run.smithi028.stdout:(3/138): ceph-immutable-object-cache-18.2.4-850 2.1 MB/s | 143 kB 00:00 2024-09-13T16:41:46.851 INFO:teuthology.orchestra.run.smithi028.stdout:(4/138): ceph-base-18.2.4-850.g50bbe7f3.el9.x86 11 MB/s | 5.1 MB 00:00 2024-09-13T16:41:46.923 INFO:teuthology.orchestra.run.smithi028.stdout:(5/138): ceph-mds-18.2.4-850.g50bbe7f3.el9.x86_ 13 MB/s | 2.1 MB 00:00 2024-09-13T16:41:46.970 INFO:teuthology.orchestra.run.smithi028.stdout:(6/138): ceph-mgr-18.2.4-850.g50bbe7f3.el9.x86_ 13 MB/s | 1.5 MB 00:00 2024-09-13T16:41:47.440 INFO:teuthology.orchestra.run.smithi028.stdout:(7/138): ceph-mon-18.2.4-850.g50bbe7f3.el9.x86_ 9.0 MB/s | 4.7 MB 00:00 2024-09-13T16:41:47.471 INFO:teuthology.orchestra.run.smithi191.stdout:(2/138): ceph-fuse-18.2.4-850.g50bbe7f3.el9.x86 1.1 MB/s | 852 kB 00:00 2024-09-13T16:41:47.580 INFO:teuthology.orchestra.run.smithi191.stdout:(3/138): ceph-immutable-object-cache-18.2.4-850 1.3 MB/s | 143 kB 00:00 2024-09-13T16:41:47.685 INFO:teuthology.orchestra.run.smithi028.stdout:(8/138): ceph-common-18.2.4-850.g50bbe7f3.el9.x 14 MB/s | 18 MB 00:01 2024-09-13T16:41:47.726 INFO:teuthology.orchestra.run.smithi028.stdout:(9/138): ceph-selinux-18.2.4-850.g50bbe7f3.el9. 613 kB/s | 25 kB 00:00 2024-09-13T16:41:47.977 INFO:teuthology.orchestra.run.smithi028.stdout:(10/138): ceph-osd-18.2.4-850.g50bbe7f3.el9.x86 16 MB/s | 17 MB 00:01 2024-09-13T16:41:48.094 INFO:teuthology.orchestra.run.smithi028.stdout:(11/138): ceph-radosgw-18.2.4-850.g50bbe7f3.el9 12 MB/s | 7.8 MB 00:00 2024-09-13T16:41:48.119 INFO:teuthology.orchestra.run.smithi028.stdout:(12/138): cephfs-mirror-18.2.4-850.g50bbe7f3.el 1.5 MB/s | 224 kB 00:00 2024-09-13T16:41:48.136 INFO:teuthology.orchestra.run.smithi028.stdout:(13/138): libcephfs-devel-18.2.4-850.g50bbe7f3. 750 kB/s | 31 kB 00:00 2024-09-13T16:41:48.195 INFO:teuthology.orchestra.run.smithi028.stdout:(14/138): libcephfs2-18.2.4-850.g50bbe7f3.el9.x 9.2 MB/s | 712 kB 00:00 2024-09-13T16:41:48.220 INFO:teuthology.orchestra.run.smithi028.stdout:(15/138): libcephsqlite-18.2.4-850.g50bbe7f3.el 1.9 MB/s | 166 kB 00:00 2024-09-13T16:41:48.245 INFO:teuthology.orchestra.run.smithi028.stdout:(16/138): librados-devel-18.2.4-850.g50bbe7f3.e 2.5 MB/s | 127 kB 00:00 2024-09-13T16:41:48.279 INFO:teuthology.orchestra.run.smithi028.stdout:(17/138): libradosstriper1-18.2.4-850.g50bbe7f3 8.0 MB/s | 477 kB 00:00 2024-09-13T16:41:48.289 INFO:teuthology.orchestra.run.smithi191.stdout:(4/138): ceph-base-18.2.4-850.g50bbe7f3.el9.x86 3.0 MB/s | 5.1 MB 00:01 2024-09-13T16:41:48.346 INFO:teuthology.orchestra.run.smithi028.stdout:(18/138): python3-ceph-argparse-18.2.4-850.g50b 675 kB/s | 45 kB 00:00 2024-09-13T16:41:48.429 INFO:teuthology.orchestra.run.smithi028.stdout:(19/138): python3-ceph-common-18.2.4-850.g50bbe 1.5 MB/s | 129 kB 00:00 2024-09-13T16:41:48.538 INFO:teuthology.orchestra.run.smithi028.stdout:(20/138): librgw2-18.2.4-850.g50bbe7f3.el9.x86_ 15 MB/s | 4.5 MB 00:00 2024-09-13T16:41:48.563 INFO:teuthology.orchestra.run.smithi028.stdout:(21/138): python3-cephfs-18.2.4-850.g50bbe7f3.e 1.2 MB/s | 162 kB 00:00 2024-09-13T16:41:48.605 INFO:teuthology.orchestra.run.smithi028.stdout:(22/138): python3-rados-18.2.4-850.g50bbe7f3.el 4.7 MB/s | 319 kB 00:00 2024-09-13T16:41:48.639 INFO:teuthology.orchestra.run.smithi028.stdout:(23/138): python3-rbd-18.2.4-850.g50bbe7f3.el9. 3.9 MB/s | 301 kB 00:00 2024-09-13T16:41:48.672 INFO:teuthology.orchestra.run.smithi028.stdout:(24/138): python3-rgw-18.2.4-850.g50bbe7f3.el9. 1.5 MB/s | 99 kB 00:00 2024-09-13T16:41:48.697 INFO:teuthology.orchestra.run.smithi191.stdout:(5/138): ceph-mds-18.2.4-850.g50bbe7f3.el9.x86_ 1.9 MB/s | 2.1 MB 00:01 2024-09-13T16:41:48.706 INFO:teuthology.orchestra.run.smithi028.stdout:(25/138): rbd-fuse-18.2.4-850.g50bbe7f3.el9.x86 1.3 MB/s | 87 kB 00:00 2024-09-13T16:41:48.765 INFO:teuthology.orchestra.run.smithi028.stdout:(26/138): rbd-nbd-18.2.4-850.g50bbe7f3.el9.x86_ 2.9 MB/s | 172 kB 00:00 2024-09-13T16:41:48.873 INFO:teuthology.orchestra.run.smithi028.stdout:(27/138): rbd-mirror-18.2.4-850.g50bbe7f3.el9.x 15 MB/s | 3.0 MB 00:00 2024-09-13T16:41:48.897 INFO:teuthology.orchestra.run.smithi191.stdout:(6/138): ceph-common-18.2.4-850.g50bbe7f3.el9.x 7.9 MB/s | 18 MB 00:02 2024-09-13T16:41:48.898 INFO:teuthology.orchestra.run.smithi028.stdout:(28/138): ceph-grafana-dashboards-18.2.4-850.g5 183 kB/s | 24 kB 00:00 2024-09-13T16:41:48.924 INFO:teuthology.orchestra.run.smithi028.stdout:(29/138): ceph-mgr-cephadm-18.2.4-850.g50bbe7f3 2.7 MB/s | 139 kB 00:00 2024-09-13T16:41:48.931 INFO:teuthology.orchestra.run.smithi191.stdout:(7/138): ceph-mgr-18.2.4-850.g50bbe7f3.el9.x86_ 2.3 MB/s | 1.5 MB 00:00 2024-09-13T16:41:49.283 INFO:teuthology.orchestra.run.smithi028.stdout:(30/138): ceph-mgr-dashboard-18.2.4-850.g50bbe7 9.2 MB/s | 3.5 MB 00:00 2024-09-13T16:41:49.374 INFO:teuthology.orchestra.run.smithi028.stdout:(31/138): ceph-mgr-modules-core-18.2.4-850.g50b 2.6 MB/s | 246 kB 00:00 2024-09-13T16:41:49.508 INFO:teuthology.orchestra.run.smithi028.stdout:(32/138): ceph-mgr-diskprediction-local-18.2.4- 13 MB/s | 7.4 MB 00:00 2024-09-13T16:41:49.558 INFO:teuthology.orchestra.run.smithi028.stdout:(33/138): ceph-mgr-rook-18.2.4-850.g50bbe7f3.el 269 kB/s | 49 kB 00:00 2024-09-13T16:41:49.609 INFO:teuthology.orchestra.run.smithi028.stdout:(34/138): ceph-prometheus-alerts-18.2.4-850.g50 164 kB/s | 16 kB 00:00 2024-09-13T16:41:49.634 INFO:teuthology.orchestra.run.smithi028.stdout:(35/138): ceph-volume-18.2.4-850.g50bbe7f3.el9. 3.4 MB/s | 264 kB 00:00 2024-09-13T16:41:49.669 INFO:teuthology.orchestra.run.smithi028.stdout:(36/138): cephadm-18.2.4-850.g50bbe7f3.el9.noar 3.8 MB/s | 226 kB 00:00 2024-09-13T16:41:49.709 INFO:teuthology.orchestra.run.smithi028.stdout:(37/138): cephfs-top-18.2.4-850.g50bbe7f3.el9.n 243 kB/s | 18 kB 00:00 2024-09-13T16:41:49.831 INFO:teuthology.orchestra.run.smithi191.stdout:(8/138): ceph-osd-18.2.4-850.g50bbe7f3.el9.x86_ 18 MB/s | 17 MB 00:00 2024-09-13T16:41:49.873 INFO:teuthology.orchestra.run.smithi191.stdout:(9/138): ceph-selinux-18.2.4-850.g50bbe7f3.el9. 606 kB/s | 25 kB 00:00 2024-09-13T16:41:50.093 INFO:teuthology.orchestra.run.smithi028.stdout:(38/138): ceph-test-18.2.4-850.g50bbe7f3.el9.x8 17 MB/s | 40 MB 00:02 2024-09-13T16:41:50.202 INFO:teuthology.orchestra.run.smithi028.stdout:(39/138): fuse-2.9.9-16.el9.x86_64.rpm 149 kB/s | 80 kB 00:00 2024-09-13T16:41:50.260 INFO:teuthology.orchestra.run.smithi028.stdout:(40/138): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-13T16:41:50.285 INFO:teuthology.orchestra.run.smithi028.stdout:(41/138): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-09-13T16:41:50.319 INFO:teuthology.orchestra.run.smithi028.stdout:(42/138): libgfortran-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 801 kB 00:00 2024-09-13T16:41:50.344 INFO:teuthology.orchestra.run.smithi028.stdout:(43/138): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-13T16:41:50.378 INFO:teuthology.orchestra.run.smithi028.stdout:(44/138): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-09-13T16:41:50.437 INFO:teuthology.orchestra.run.smithi028.stdout:(45/138): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-09-13T16:41:50.462 INFO:teuthology.orchestra.run.smithi028.stdout:(46/138): python3-urllib3-1.26.5-6.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-09-13T16:41:50.495 INFO:teuthology.orchestra.run.smithi028.stdout:(47/138): python3-cryptography-36.0.1-4.el9.x86 5.2 MB/s | 1.2 MB 00:00 2024-09-13T16:41:50.596 INFO:teuthology.orchestra.run.smithi028.stdout:(48/138): boost-program-options-1.75.0-8.el9.x8 1.0 MB/s | 107 kB 00:00 2024-09-13T16:41:50.629 INFO:teuthology.orchestra.run.smithi028.stdout:(49/138): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 689 kB 00:00 2024-09-13T16:41:50.679 INFO:teuthology.orchestra.run.smithi028.stdout:(50/138): elfutils-libelf-devel-0.191-4.el9.x86 260 kB/s | 22 kB 00:00 2024-09-13T16:41:50.713 INFO:teuthology.orchestra.run.smithi028.stdout:(51/138): flex-2.6.4-9.el9.x86_64.rpm 3.7 MB/s | 313 kB 00:00 2024-09-13T16:41:50.763 INFO:teuthology.orchestra.run.smithi028.stdout:(52/138): flexiblas-3.0.4-8.el9.x86_64.rpm 392 kB/s | 33 kB 00:00 2024-09-13T16:41:50.814 INFO:teuthology.orchestra.run.smithi028.stdout:(53/138): bison-3.7.4-5.el9.x86_64.rpm 2.6 MB/s | 943 kB 00:00 2024-09-13T16:41:50.856 INFO:teuthology.orchestra.run.smithi028.stdout:(54/138): flexiblas-openblas-openmp-3.0.4-8.el9 198 kB/s | 18 kB 00:00 2024-09-13T16:41:50.906 INFO:teuthology.orchestra.run.smithi028.stdout:(55/138): libaio-devel-0.3.111-13.el9.x86_64.rp 121 kB/s | 11 kB 00:00 2024-09-13T16:41:50.939 INFO:teuthology.orchestra.run.smithi028.stdout:(56/138): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-09-13T16:41:50.998 INFO:teuthology.orchestra.run.smithi028.stdout:(57/138): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-09-13T16:41:51.032 INFO:teuthology.orchestra.run.smithi191.stdout:(10/138): ceph-test-18.2.4-850.g50bbe7f3.el9.x8 34 MB/s | 40 MB 00:01 2024-09-13T16:41:51.056 INFO:teuthology.orchestra.run.smithi028.stdout:(58/138): librabbitmq-0.11.0-7.el9.x86_64.rpm 301 kB/s | 45 kB 00:00 2024-09-13T16:41:51.082 INFO:teuthology.orchestra.run.smithi191.stdout:(11/138): cephfs-mirror-18.2.4-850.g50bbe7f3.el 4.4 MB/s | 224 kB 00:00 2024-09-13T16:41:51.090 INFO:teuthology.orchestra.run.smithi028.stdout:(59/138): libtool-2.4.6-46.el9.x86_64.rpm 6.2 MB/s | 581 kB 00:00 2024-09-13T16:41:51.116 INFO:teuthology.orchestra.run.smithi028.stdout:(60/138): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-09-13T16:41:51.157 INFO:teuthology.orchestra.run.smithi028.stdout:(61/138): libuuid-devel-2.37.4-20.el9.x86_64.rp 244 kB/s | 24 kB 00:00 2024-09-13T16:41:51.158 INFO:teuthology.orchestra.run.smithi191.stdout:(12/138): ceph-mon-18.2.4-850.g50bbe7f3.el9.x86 1.9 MB/s | 4.7 MB 00:02 2024-09-13T16:41:51.183 INFO:teuthology.orchestra.run.smithi191.stdout:(13/138): libcephfs-devel-18.2.4-850.g50bbe7f3. 310 kB/s | 31 kB 00:00 2024-09-13T16:41:51.199 INFO:teuthology.orchestra.run.smithi028.stdout:(62/138): libzstd-devel-1.5.1-2.el9.x86_64.rpm 437 kB/s | 47 kB 00:00 2024-09-13T16:41:51.224 INFO:teuthology.orchestra.run.smithi028.stdout:(63/138): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-13T16:41:51.233 INFO:teuthology.orchestra.run.smithi191.stdout:(14/138): libcephsqlite-18.2.4-850.g50bbe7f3.el 3.3 MB/s | 166 kB 00:00 2024-09-13T16:41:51.267 INFO:teuthology.orchestra.run.smithi028.stdout:(64/138): openblas-0.3.26-2.el9.x86_64.rpm 353 kB/s | 38 kB 00:00 2024-09-13T16:41:51.325 INFO:teuthology.orchestra.run.smithi191.stdout:(15/138): ceph-radosgw-18.2.4-850.g50bbe7f3.el9 3.2 MB/s | 7.8 MB 00:02 2024-09-13T16:41:51.350 INFO:teuthology.orchestra.run.smithi191.stdout:(16/138): librados-devel-18.2.4-850.g50bbe7f3.e 1.1 MB/s | 127 kB 00:00 2024-09-13T16:41:51.375 INFO:teuthology.orchestra.run.smithi191.stdout:(17/138): libcephfs2-18.2.4-850.g50bbe7f3.el9.x 3.2 MB/s | 712 kB 00:00 2024-09-13T16:41:51.392 INFO:teuthology.orchestra.run.smithi028.stdout:(65/138): openssl-devel-3.2.2-6.el9.x86_64.rpm 27 MB/s | 4.4 MB 00:00 2024-09-13T16:41:51.425 INFO:teuthology.orchestra.run.smithi191.stdout:(18/138): libradosstriper1-18.2.4-850.g50bbe7f3 4.6 MB/s | 477 kB 00:00 2024-09-13T16:41:51.451 INFO:teuthology.orchestra.run.smithi191.stdout:(19/138): python3-ceph-argparse-18.2.4-850.g50b 598 kB/s | 45 kB 00:00 2024-09-13T16:41:51.475 INFO:teuthology.orchestra.run.smithi028.stdout:(66/138): openblas-openmp-0.3.26-2.el9.x86_64.r 18 MB/s | 4.9 MB 00:00 2024-09-13T16:41:51.476 INFO:teuthology.orchestra.run.smithi191.stdout:(20/138): python3-ceph-common-18.2.4-850.g50bbe 2.5 MB/s | 129 kB 00:00 2024-09-13T16:41:51.517 INFO:teuthology.orchestra.run.smithi028.stdout:(67/138): postgresql-13.16-1.el9.x86_64.rpm 6.4 MB/s | 1.6 MB 00:00 2024-09-13T16:41:51.551 INFO:teuthology.orchestra.run.smithi028.stdout:(68/138): postgresql-contrib-13.16-1.el9.x86_64 5.6 MB/s | 910 kB 00:00 2024-09-13T16:41:51.551 INFO:teuthology.orchestra.run.smithi191.stdout:(21/138): librgw2-18.2.4-850.g50bbe7f3.el9.x86_ 22 MB/s | 4.5 MB 00:00 2024-09-13T16:41:51.576 INFO:teuthology.orchestra.run.smithi028.stdout:(69/138): postgresql-private-libs-13.16-1.el9.x 1.3 MB/s | 137 kB 00:00 2024-09-13T16:41:51.576 INFO:teuthology.orchestra.run.smithi191.stdout:(22/138): python3-cephfs-18.2.4-850.g50bbe7f3.e 1.3 MB/s | 162 kB 00:00 2024-09-13T16:41:51.602 INFO:teuthology.orchestra.run.smithi191.stdout:(23/138): python3-rados-18.2.4-850.g50bbe7f3.el 2.5 MB/s | 319 kB 00:00 2024-09-13T16:41:51.627 INFO:teuthology.orchestra.run.smithi191.stdout:(24/138): python3-rbd-18.2.4-850.g50bbe7f3.el9. 3.9 MB/s | 301 kB 00:00 2024-09-13T16:41:51.652 INFO:teuthology.orchestra.run.smithi191.stdout:(25/138): python3-rgw-18.2.4-850.g50bbe7f3.el9. 1.3 MB/s | 99 kB 00:00 2024-09-13T16:41:51.660 INFO:teuthology.orchestra.run.smithi028.stdout:(70/138): python3-devel-3.9.19-8.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-09-13T16:41:51.678 INFO:teuthology.orchestra.run.smithi191.stdout:(26/138): rbd-fuse-18.2.4-850.g50bbe7f3.el9.x86 1.1 MB/s | 87 kB 00:00 2024-09-13T16:41:51.754 INFO:teuthology.orchestra.run.smithi191.stdout:(27/138): rbd-mirror-18.2.4-850.g50bbe7f3.el9.x 24 MB/s | 3.0 MB 00:00 2024-09-13T16:41:51.768 INFO:teuthology.orchestra.run.smithi028.stdout:(71/138): python3-babel-2.9.1-2.el9.noarch.rpm 28 MB/s | 6.0 MB 00:00 2024-09-13T16:41:51.770 INFO:teuthology.orchestra.run.smithi191.stdout:(28/138): ceph-grafana-dashboards-18.2.4-850.g5 264 kB/s | 24 kB 00:00 2024-09-13T16:41:51.793 INFO:teuthology.orchestra.run.smithi028.stdout:(72/138): python3-jinja2-2.11.3-6.el9.noarch.rp 1.8 MB/s | 248 kB 00:00 2024-09-13T16:41:51.795 INFO:teuthology.orchestra.run.smithi191.stdout:(29/138): rbd-nbd-18.2.4-850.g50bbe7f3.el9.x86_ 1.2 MB/s | 172 kB 00:00 2024-09-13T16:41:51.821 INFO:teuthology.orchestra.run.smithi191.stdout:(30/138): ceph-mgr-cephadm-18.2.4-850.g50bbe7f3 2.0 MB/s | 139 kB 00:00 2024-09-13T16:41:51.869 INFO:teuthology.orchestra.run.smithi028.stdout:(73/138): postgresql-server-13.16-1.el9.x86_64. 16 MB/s | 5.8 MB 00:00 2024-09-13T16:41:51.871 INFO:teuthology.orchestra.run.smithi191.stdout:(31/138): ceph-mgr-modules-core-18.2.4-850.g50b 4.8 MB/s | 246 kB 00:00 2024-09-13T16:41:51.914 INFO:teuthology.orchestra.run.smithi028.stdout:(74/138): python3-jsonpatch-1.21-16.el9.noarch. 181 kB/s | 26 kB 00:00 2024-09-13T16:41:51.914 INFO:teuthology.orchestra.run.smithi191.stdout:(32/138): ceph-mgr-rook-18.2.4-850.g50bbe7f3.el 1.2 MB/s | 49 kB 00:00 2024-09-13T16:41:51.953 INFO:teuthology.orchestra.run.smithi028.stdout:(75/138): python3-jsonpointer-2.0-4.el9.noarch. 121 kB/s | 19 kB 00:00 2024-09-13T16:41:51.955 INFO:teuthology.orchestra.run.smithi191.stdout:(33/138): ceph-prometheus-alerts-18.2.4-850.g50 396 kB/s | 16 kB 00:00 2024-09-13T16:41:51.977 INFO:teuthology.orchestra.run.smithi028.stdout:(76/138): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-09-13T16:41:52.005 INFO:teuthology.orchestra.run.smithi191.stdout:(34/138): ceph-volume-18.2.4-850.g50bbe7f3.el9. 5.2 MB/s | 264 kB 00:00 2024-09-13T16:41:52.045 INFO:teuthology.orchestra.run.smithi028.stdout:(77/138): python3-markupsafe-1.1.1-12.el9.x86_6 379 kB/s | 35 kB 00:00 2024-09-13T16:41:52.055 INFO:teuthology.orchestra.run.smithi191.stdout:(35/138): cephadm-18.2.4-850.g50bbe7f3.el9.noar 4.4 MB/s | 226 kB 00:00 2024-09-13T16:41:52.087 INFO:teuthology.orchestra.run.smithi028.stdout:(78/138): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.2 MB/s | 250 kB 00:00 2024-09-13T16:41:52.097 INFO:teuthology.orchestra.run.smithi191.stdout:(36/138): cephfs-top-18.2.4-850.g50bbe7f3.el9.n 440 kB/s | 18 kB 00:00 2024-09-13T16:41:52.154 INFO:teuthology.orchestra.run.smithi028.stdout:(79/138): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 5.1 MB 00:00 2024-09-13T16:41:52.179 INFO:teuthology.orchestra.run.smithi028.stdout:(80/138): python3-packaging-20.9-5.el9.noarch.r 842 kB/s | 77 kB 00:00 2024-09-13T16:41:52.204 INFO:teuthology.orchestra.run.smithi028.stdout:(81/138): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 222 kB 00:00 2024-09-13T16:41:52.231 INFO:teuthology.orchestra.run.smithi191.stdout:(37/138): fuse-2.9.9-16.el9.x86_64.rpm 599 kB/s | 80 kB 00:00 2024-09-13T16:41:52.246 INFO:teuthology.orchestra.run.smithi028.stdout:(82/138): python3-prettytable-0.7.2-27.el9.noar 459 kB/s | 42 kB 00:00 2024-09-13T16:41:52.271 INFO:teuthology.orchestra.run.smithi028.stdout:(83/138): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-09-13T16:41:52.306 INFO:teuthology.orchestra.run.smithi191.stdout:(38/138): ceph-mgr-dashboard-18.2.4-850.g50bbe7 6.6 MB/s | 3.5 MB 00:00 2024-09-13T16:41:52.323 INFO:teuthology.orchestra.run.smithi028.stdout:(84/138): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 279 kB 00:00 2024-09-13T16:41:52.372 INFO:teuthology.orchestra.run.smithi028.stdout:(85/138): python3-pytz-2021.1-5.el9.noarch.rpm 407 kB/s | 51 kB 00:00 2024-09-13T16:41:52.413 INFO:teuthology.orchestra.run.smithi028.stdout:(86/138): python3-requests-oauthlib-1.3.0-12.el 378 kB/s | 54 kB 00:00 2024-09-13T16:41:52.473 INFO:teuthology.orchestra.run.smithi191.stdout:(39/138): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-13T16:41:52.489 INFO:teuthology.orchestra.run.smithi028.stdout:(87/138): socat-1.7.4.1-6.el9.x86_64.rpm 4.0 MB/s | 304 kB 00:00 2024-09-13T16:41:52.498 INFO:teuthology.orchestra.run.smithi191.stdout:(40/138): libgfortran-11.5.0-2.el9.x86_64.rpm 2.9 MB/s | 801 kB 00:00 2024-09-13T16:41:52.531 INFO:teuthology.orchestra.run.smithi028.stdout:(88/138): python3-toml-0.10.2-6.el9.noarch.rpm 356 kB/s | 42 kB 00:00 2024-09-13T16:41:52.573 INFO:teuthology.orchestra.run.smithi028.stdout:(89/138): uuid-1.6.2-55.el9.x86_64.rpm 694 kB/s | 58 kB 00:00 2024-09-13T16:41:52.582 INFO:teuthology.orchestra.run.smithi191.stdout:(41/138): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 253 kB 00:00 2024-09-13T16:41:52.606 INFO:teuthology.orchestra.run.smithi028.stdout:(90/138): xfsprogs-devel-6.4.0-4.el9.x86_64.rpm 1.2 MB/s | 96 kB 00:00 2024-09-13T16:41:52.632 INFO:teuthology.orchestra.run.smithi191.stdout:(42/138): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-13T16:41:52.665 INFO:teuthology.orchestra.run.smithi028.stdout:(91/138): xmlstarlet-1.6.1-20.el9.x86_64.rpm 692 kB/s | 64 kB 00:00 2024-09-13T16:41:52.700 INFO:teuthology.orchestra.run.smithi191.stdout:(43/138): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-09-13T16:41:52.707 INFO:teuthology.orchestra.run.smithi028.stdout:(92/138): zlib-devel-1.2.11-41.el9.x86_64.rpm 447 kB/s | 45 kB 00:00 2024-09-13T16:41:52.757 INFO:teuthology.orchestra.run.smithi028.stdout:(93/138): fmt-8.1.1-5.el9.x86_64.rpm 1.2 MB/s | 111 kB 00:00 2024-09-13T16:41:52.783 INFO:teuthology.orchestra.run.smithi191.stdout:(44/138): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-09-13T16:41:52.807 INFO:teuthology.orchestra.run.smithi028.stdout:(94/138): gperftools-libs-2.9.1-3.el9.x86_64.rp 3.0 MB/s | 308 kB 00:00 2024-09-13T16:41:52.825 INFO:teuthology.orchestra.run.smithi191.stdout:(45/138): python3-cryptography-36.0.1-4.el9.x86 3.8 MB/s | 1.2 MB 00:00 2024-09-13T16:41:52.849 INFO:teuthology.orchestra.run.smithi028.stdout:(95/138): libarrow-doc-9.0.0-11.el9.noarch.rpm 628 kB/s | 26 kB 00:00 2024-09-13T16:41:52.875 INFO:teuthology.orchestra.run.smithi191.stdout:(46/138): python3-urllib3-1.26.5-6.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-09-13T16:41:52.891 INFO:teuthology.orchestra.run.smithi028.stdout:(96/138): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-09-13T16:41:52.925 INFO:teuthology.orchestra.run.smithi028.stdout:(97/138): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-13T16:41:53.034 INFO:teuthology.orchestra.run.smithi191.stdout:(47/138): automake-1.16.2-8.el9.noarch.rpm 3.2 MB/s | 689 kB 00:00 2024-09-13T16:41:53.084 INFO:teuthology.orchestra.run.smithi191.stdout:(48/138): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-09-13T16:41:53.134 INFO:teuthology.orchestra.run.smithi191.stdout:(49/138): elfutils-libelf-devel-0.191-4.el9.x86 434 kB/s | 22 kB 00:00 2024-09-13T16:41:53.142 INFO:teuthology.orchestra.run.smithi028.stdout:(98/138): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-09-13T16:41:53.160 INFO:teuthology.orchestra.run.smithi191.stdout:(50/138): bison-3.7.4-5.el9.x86_64.rpm 3.2 MB/s | 943 kB 00:00 2024-09-13T16:41:53.200 INFO:teuthology.orchestra.run.smithi028.stdout:(99/138): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.0 MB/s | 839 kB 00:00 2024-09-13T16:41:53.250 INFO:teuthology.orchestra.run.smithi028.stdout:(100/138): python3-bcrypt-3.2.2-1.el9.x86_64.rp 914 kB/s | 43 kB 00:00 2024-09-13T16:41:53.285 INFO:teuthology.orchestra.run.smithi191.stdout:(51/138): ceph-mgr-diskprediction-local-18.2.4- 5.0 MB/s | 7.4 MB 00:01 2024-09-13T16:41:53.293 INFO:teuthology.orchestra.run.smithi028.stdout:(101/138): python3-cachetools-4.2.4-1.el9.noarc 770 kB/s | 32 kB 00:00 2024-09-13T16:41:53.310 INFO:teuthology.orchestra.run.smithi191.stdout:(52/138): flexiblas-3.0.4-8.el9.x86_64.rpm 218 kB/s | 33 kB 00:00 2024-09-13T16:41:53.334 INFO:teuthology.orchestra.run.smithi028.stdout:(102/138): python3-certifi-2023.05.07-4.el9.noa 342 kB/s | 14 kB 00:00 2024-09-13T16:41:53.344 INFO:teuthology.orchestra.run.smithi191.stdout:(53/138): flex-2.6.4-9.el9.x86_64.rpm 1.5 MB/s | 313 kB 00:00 2024-09-13T16:41:53.368 INFO:teuthology.orchestra.run.smithi028.stdout:(103/138): python3-cheroot-8.6.0-4.el9.noarch.r 5.0 MB/s | 172 kB 00:00 2024-09-13T16:41:53.369 INFO:teuthology.orchestra.run.smithi191.stdout:(54/138): flexiblas-openblas-openmp-3.0.4-8.el9 311 kB/s | 18 kB 00:00 2024-09-13T16:41:53.395 INFO:teuthology.orchestra.run.smithi191.stdout:(55/138): libaio-devel-0.3.111-13.el9.x86_64.rp 221 kB/s | 11 kB 00:00 2024-09-13T16:41:53.402 INFO:teuthology.orchestra.run.smithi028.stdout:(104/138): python3-asyncssh-2.13.2-5.el9.noarch 2.1 MB/s | 548 kB 00:00 2024-09-13T16:41:53.435 INFO:teuthology.orchestra.run.smithi028.stdout:(105/138): python3-cherrypy-18.6.1-2.el9.noarch 5.3 MB/s | 358 kB 00:00 2024-09-13T16:41:53.445 INFO:teuthology.orchestra.run.smithi191.stdout:(56/138): librabbitmq-0.11.0-7.el9.x86_64.rpm 907 kB/s | 45 kB 00:00 2024-09-13T16:41:53.460 INFO:teuthology.orchestra.run.smithi028.stdout:(106/138): python3-google-auth-2.34.0-1.el9.noa 3.7 MB/s | 220 kB 00:00 2024-09-13T16:41:53.470 INFO:teuthology.orchestra.run.smithi191.stdout:(57/138): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2024-09-13T16:41:53.502 INFO:teuthology.orchestra.run.smithi028.stdout:(107/138): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-09-13T16:41:53.545 INFO:teuthology.orchestra.run.smithi028.stdout:(108/138): python3-jaraco-classes-3.2.1-5.el9.n 212 kB/s | 18 kB 00:00 2024-09-13T16:41:53.612 INFO:teuthology.orchestra.run.smithi191.stdout:(58/138): librdkafka-1.6.1-102.el9.x86_64.rpm 3.9 MB/s | 662 kB 00:00 2024-09-13T16:41:53.628 INFO:teuthology.orchestra.run.smithi028.stdout:(109/138): python3-jaraco-collections-3.0.0-8.e 184 kB/s | 23 kB 00:00 2024-09-13T16:41:53.637 INFO:teuthology.orchestra.run.smithi191.stdout:(59/138): libtool-2.4.6-46.el9.x86_64.rpm 3.4 MB/s | 581 kB 00:00 2024-09-13T16:41:53.662 INFO:teuthology.orchestra.run.smithi191.stdout:(60/138): libuuid-devel-2.37.4-20.el9.x86_64.rp 489 kB/s | 24 kB 00:00 2024-09-13T16:41:53.670 INFO:teuthology.orchestra.run.smithi028.stdout:(110/138): python3-jaraco-functools-3.5.0-2.el9 155 kB/s | 19 kB 00:00 2024-09-13T16:41:53.688 INFO:teuthology.orchestra.run.smithi191.stdout:(61/138): libzstd-devel-1.5.1-2.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-09-13T16:41:53.712 INFO:teuthology.orchestra.run.smithi028.stdout:(111/138): python3-jaraco-text-3.2.0-6.el9.noar 236 kB/s | 20 kB 00:00 2024-09-13T16:41:53.738 INFO:teuthology.orchestra.run.smithi191.stdout:(62/138): openblas-0.3.26-2.el9.x86_64.rpm 771 kB/s | 38 kB 00:00 2024-09-13T16:41:53.754 INFO:teuthology.orchestra.run.smithi028.stdout:(112/138): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-09-13T16:41:53.772 INFO:teuthology.orchestra.run.smithi191.stdout:(63/138): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-13T16:41:53.821 INFO:teuthology.orchestra.run.smithi028.stdout:(113/138): libarrow-9.0.0-11.el9.x86_64.rpm 4.2 MB/s | 4.4 MB 00:01 2024-09-13T16:41:53.846 INFO:teuthology.orchestra.run.smithi028.stdout:(114/138): python3-more-itertools-8.12.0-2.el9. 859 kB/s | 79 kB 00:00 2024-09-13T16:41:53.888 INFO:teuthology.orchestra.run.smithi028.stdout:(115/138): python3-natsort-7.1.1-5.el9.noarch.r 865 kB/s | 58 kB 00:00 2024-09-13T16:41:53.936 INFO:teuthology.orchestra.run.smithi028.stdout:(116/138): python3-portend-3.1.0-2.el9.noarch.r 346 kB/s | 16 kB 00:00 2024-09-13T16:41:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:(117/138): python3-pyOpenSSL-21.0.0-1.el9.noarc 5.3 MB/s | 90 kB 00:00 2024-09-13T16:41:54.006 INFO:teuthology.orchestra.run.smithi028.stdout:(118/138): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-13T16:41:54.037 INFO:teuthology.orchestra.run.smithi028.stdout:(119/138): python3-repoze-lru-0.7-16.el9.noarch 369 kB/s | 31 kB 00:00 2024-09-13T16:41:54.079 INFO:teuthology.orchestra.run.smithi028.stdout:(120/138): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-09-13T16:41:54.105 INFO:teuthology.orchestra.run.smithi028.stdout:(121/138): python3-routes-2.5.1-5.el9.noarch.rp 1.9 MB/s | 188 kB 00:00 2024-09-13T16:41:54.147 INFO:teuthology.orchestra.run.smithi028.stdout:(122/138): python3-tempora-5.0.0-2.el9.noarch.r 536 kB/s | 36 kB 00:00 2024-09-13T16:41:54.172 INFO:teuthology.orchestra.run.smithi028.stdout:(123/138): python3-typing-extensions-4.12.2-2.e 1.1 MB/s | 74 kB 00:00 2024-09-13T16:41:54.214 INFO:teuthology.orchestra.run.smithi028.stdout:(124/138): python3-kubernetes-26.1.0-2.el9.noar 1.9 MB/s | 1.0 MB 00:00 2024-09-13T16:41:54.239 INFO:teuthology.orchestra.run.smithi191.stdout:(64/138): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.1 MB/s | 3.0 MB 00:00 2024-09-13T16:41:54.256 INFO:teuthology.orchestra.run.smithi028.stdout:(125/138): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-09-13T16:41:54.281 INFO:teuthology.orchestra.run.smithi028.stdout:(126/138): python3-webob-1.8.8-2.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-09-13T16:41:54.323 INFO:teuthology.orchestra.run.smithi028.stdout:(127/138): python3-zc-lockfile-2.0-10.el9.noarc 300 kB/s | 20 kB 00:00 2024-09-13T16:41:54.356 INFO:teuthology.orchestra.run.smithi028.stdout:(128/138): re2-20211101-20.el9.x86_64.rpm 2.5 MB/s | 191 kB 00:00 2024-09-13T16:41:54.382 INFO:teuthology.orchestra.run.smithi028.stdout:(129/138): python3-werkzeug-2.0.3-3.el9.1.noarc 2.5 MB/s | 427 kB 00:00 2024-09-13T16:41:54.457 INFO:teuthology.orchestra.run.smithi028.stdout:(130/138): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-09-13T16:41:54.580 INFO:teuthology.orchestra.run.smithi191.stdout:(65/138): postgresql-13.16-1.el9.x86_64.rpm 4.6 MB/s | 1.6 MB 00:00 2024-09-13T16:41:54.641 INFO:teuthology.orchestra.run.smithi028.stdout:(131/138): librados2-18.2.4-850.g50bbe7f3.el9.x 12 MB/s | 3.3 MB 00:00 2024-09-13T16:41:54.707 INFO:teuthology.orchestra.run.smithi028.stdout:(132/138): librbd1-18.2.4-850.g50bbe7f3.el9.x86 9.3 MB/s | 3.0 MB 00:00 2024-09-13T16:41:54.722 INFO:teuthology.orchestra.run.smithi191.stdout:(66/138): openssl-devel-3.2.2-6.el9.x86_64.rpm 4.7 MB/s | 4.4 MB 00:00 2024-09-13T16:41:54.773 INFO:teuthology.orchestra.run.smithi191.stdout:(67/138): postgresql-private-libs-13.16-1.el9.x 2.7 MB/s | 137 kB 00:00 2024-09-13T16:41:54.833 INFO:teuthology.orchestra.run.smithi028.stdout:(133/138): NetworkManager-team-1.49.90-1.el9.x8 326 kB/s | 41 kB 00:00 2024-09-13T16:41:54.856 INFO:teuthology.orchestra.run.smithi191.stdout:(68/138): openblas-openmp-0.3.26-2.el9.x86_64.r 4.4 MB/s | 4.9 MB 00:01 2024-09-13T16:41:54.890 INFO:teuthology.orchestra.run.smithi191.stdout:(69/138): postgresql-contrib-13.16-1.el9.x86_64 2.9 MB/s | 910 kB 00:00 2024-09-13T16:41:54.914 INFO:teuthology.orchestra.run.smithi028.stdout:(134/138): NetworkManager-1.49.90-1.el9.x86_64. 5.1 MB/s | 2.3 MB 00:00 2024-09-13T16:41:54.950 INFO:teuthology.orchestra.run.smithi028.stdout:(135/138): NetworkManager-tui-1.49.90-1.el9.x86 2.1 MB/s | 249 kB 00:00 2024-09-13T16:41:54.974 INFO:teuthology.orchestra.run.smithi191.stdout:(70/138): python3-devel-3.9.19-8.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-09-13T16:41:55.025 INFO:teuthology.orchestra.run.smithi028.stdout:(136/138): NetworkManager-libnm-1.49.90-1.el9.x 4.7 MB/s | 1.8 MB 00:00 2024-09-13T16:41:55.057 INFO:teuthology.orchestra.run.smithi191.stdout:(71/138): python3-jinja2-2.11.3-6.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-09-13T16:41:55.084 INFO:teuthology.orchestra.run.smithi028.stdout:(137/138): openssl-3.2.2-6.el9.x86_64.rpm 8.0 MB/s | 1.4 MB 00:00 2024-09-13T16:41:55.107 INFO:teuthology.orchestra.run.smithi191.stdout:(72/138): python3-jsonpatch-1.21-16.el9.noarch. 514 kB/s | 26 kB 00:00 2024-09-13T16:41:55.158 INFO:teuthology.orchestra.run.smithi191.stdout:(73/138): python3-jsonpointer-2.0-4.el9.noarch. 380 kB/s | 19 kB 00:00 2024-09-13T16:41:55.217 INFO:teuthology.orchestra.run.smithi028.stdout:(138/138): openssl-libs-3.2.2-6.el9.x86_64.rpm 9.0 MB/s | 2.4 MB 00:00 2024-09-13T16:41:55.223 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-13T16:41:55.223 INFO:teuthology.orchestra.run.smithi028.stdout:Total 23 MB/s | 203 MB 00:08 2024-09-13T16:41:55.241 INFO:teuthology.orchestra.run.smithi191.stdout:(74/138): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-13T16:41:55.300 INFO:teuthology.orchestra.run.smithi191.stdout:(75/138): python3-markupsafe-1.1.1-12.el9.x86_6 596 kB/s | 35 kB 00:00 2024-09-13T16:41:55.767 INFO:teuthology.orchestra.run.smithi191.stdout:(76/138): postgresql-server-13.16-1.el9.x86_64. 5.8 MB/s | 5.8 MB 00:00 2024-09-13T16:41:55.850 INFO:teuthology.orchestra.run.smithi191.stdout:(77/138): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.0 MB/s | 250 kB 00:00 2024-09-13T16:41:56.001 INFO:teuthology.orchestra.run.smithi191.stdout:(78/138): python3-babel-2.9.1-2.el9.noarch.rpm 5.2 MB/s | 6.0 MB 00:01 2024-09-13T16:41:56.026 INFO:teuthology.orchestra.run.smithi191.stdout:(79/138): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 222 kB 00:00 2024-09-13T16:41:56.051 INFO:teuthology.orchestra.run.smithi191.stdout:(80/138): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-09-13T16:41:56.076 INFO:teuthology.orchestra.run.smithi191.stdout:(81/138): python3-prettytable-0.7.2-27.el9.noar 843 kB/s | 42 kB 00:00 2024-09-13T16:41:56.102 INFO:teuthology.orchestra.run.smithi191.stdout:(82/138): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-09-13T16:41:56.152 INFO:teuthology.orchestra.run.smithi191.stdout:(83/138): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-13T16:41:56.177 INFO:teuthology.orchestra.run.smithi191.stdout:(84/138): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 279 kB 00:00 2024-09-13T16:41:56.203 INFO:teuthology.orchestra.run.smithi191.stdout:(85/138): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-09-13T16:41:56.253 INFO:teuthology.orchestra.run.smithi191.stdout:(86/138): python3-toml-0.10.2-6.el9.noarch.rpm 833 kB/s | 42 kB 00:00 2024-09-13T16:41:56.336 INFO:teuthology.orchestra.run.smithi191.stdout:(87/138): socat-1.7.4.1-6.el9.x86_64.rpm 3.6 MB/s | 304 kB 00:00 2024-09-13T16:41:56.395 INFO:teuthology.orchestra.run.smithi191.stdout:(88/138): uuid-1.6.2-55.el9.x86_64.rpm 989 kB/s | 58 kB 00:00 2024-09-13T16:41:56.487 INFO:teuthology.orchestra.run.smithi191.stdout:(89/138): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.3 MB/s | 5.1 MB 00:01 2024-09-13T16:41:56.512 INFO:teuthology.orchestra.run.smithi191.stdout:(90/138): xfsprogs-devel-6.4.0-4.el9.x86_64.rpm 820 kB/s | 96 kB 00:00 2024-09-13T16:41:56.537 INFO:teuthology.orchestra.run.smithi191.stdout:(91/138): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-13T16:41:56.571 INFO:teuthology.orchestra.run.smithi191.stdout:(92/138): zlib-devel-1.2.11-41.el9.x86_64.rpm 764 kB/s | 45 kB 00:00 2024-09-13T16:41:56.605 INFO:teuthology.orchestra.run.smithi191.stdout:(93/138): fmt-8.1.1-5.el9.x86_64.rpm 1.6 MB/s | 111 kB 00:00 2024-09-13T16:41:56.655 INFO:teuthology.orchestra.run.smithi191.stdout:(94/138): gperftools-libs-2.9.1-3.el9.x86_64.rp 3.7 MB/s | 308 kB 00:00 2024-09-13T16:41:56.680 INFO:teuthology.orchestra.run.smithi191.stdout:(95/138): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-13T16:41:56.713 INFO:teuthology.orchestra.run.smithi191.stdout:(96/138): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-13T16:41:56.747 INFO:teuthology.orchestra.run.smithi191.stdout:(97/138): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-13T16:41:56.831 INFO:teuthology.orchestra.run.smithi191.stdout:(98/138): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 839 kB 00:00 2024-09-13T16:41:56.890 INFO:teuthology.orchestra.run.smithi191.stdout:(99/138): python3-asyncssh-2.13.2-5.el9.noarch. 9.2 MB/s | 548 kB 00:00 2024-09-13T16:41:56.906 INFO:teuthology.orchestra.run.smithi191.stdout:(100/138): python3-bcrypt-3.2.2-1.el9.x86_64.rp 2.6 MB/s | 43 kB 00:00 2024-09-13T16:41:56.965 INFO:teuthology.orchestra.run.smithi191.stdout:(101/138): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-13T16:41:56.982 INFO:teuthology.orchestra.run.smithi191.stdout:(102/138): python3-cachetools-4.2.4-1.el9.noarc 428 kB/s | 32 kB 00:00 2024-09-13T16:41:57.008 INFO:teuthology.orchestra.run.smithi191.stdout:(103/138): python3-certifi-2023.05.07-4.el9.noa 334 kB/s | 14 kB 00:00 2024-09-13T16:41:57.041 INFO:teuthology.orchestra.run.smithi191.stdout:(104/138): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-09-13T16:41:57.052 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-13T16:41:57.066 INFO:teuthology.orchestra.run.smithi191.stdout:(105/138): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-09-13T16:41:57.092 INFO:teuthology.orchestra.run.smithi191.stdout:(106/138): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-09-13T16:41:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:(107/138): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-09-13T16:41:57.142 INFO:teuthology.orchestra.run.smithi191.stdout:(108/138): python3-jaraco-classes-3.2.1-5.el9.n 351 kB/s | 18 kB 00:00 2024-09-13T16:41:57.168 INFO:teuthology.orchestra.run.smithi191.stdout:(109/138): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-09-13T16:41:57.193 INFO:teuthology.orchestra.run.smithi191.stdout:(110/138): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-09-13T16:41:57.218 INFO:teuthology.orchestra.run.smithi191.stdout:(111/138): python3-jaraco-text-3.2.0-6.el9.noar 394 kB/s | 20 kB 00:00 2024-09-13T16:41:57.241 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-13T16:41:57.241 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-13T16:41:57.243 INFO:teuthology.orchestra.run.smithi191.stdout:(112/138): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-13T16:41:57.277 INFO:teuthology.orchestra.run.smithi191.stdout:(113/138): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-09-13T16:41:57.311 INFO:teuthology.orchestra.run.smithi191.stdout:(114/138): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-09-13T16:41:57.370 INFO:teuthology.orchestra.run.smithi191.stdout:(115/138): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-09-13T16:41:57.370 INFO:teuthology.orchestra.run.smithi191.stdout:(116/138): python3-portend-3.1.0-2.el9.noarch.r 658 kB/s | 16 kB 00:00 2024-09-13T16:41:57.395 INFO:teuthology.orchestra.run.smithi191.stdout:(117/138): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-09-13T16:41:57.420 INFO:teuthology.orchestra.run.smithi191.stdout:(118/138): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-09-13T16:41:57.446 INFO:teuthology.orchestra.run.smithi191.stdout:(119/138): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-09-13T16:41:57.471 INFO:teuthology.orchestra.run.smithi191.stdout:(120/138): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-13T16:41:57.496 INFO:teuthology.orchestra.run.smithi191.stdout:(121/138): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-09-13T16:41:57.521 INFO:teuthology.orchestra.run.smithi191.stdout:(122/138): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2024-09-13T16:41:57.546 INFO:teuthology.orchestra.run.smithi191.stdout:(123/138): python3-typing-extensions-4.12.2-2.e 1.4 MB/s | 74 kB 00:00 2024-09-13T16:41:57.572 INFO:teuthology.orchestra.run.smithi191.stdout:(124/138): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-09-13T16:41:57.597 INFO:teuthology.orchestra.run.smithi191.stdout:(125/138): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-13T16:41:57.622 INFO:teuthology.orchestra.run.smithi191.stdout:(126/138): python3-zc-lockfile-2.0-10.el9.noarc 803 kB/s | 20 kB 00:00 2024-09-13T16:41:57.647 INFO:teuthology.orchestra.run.smithi191.stdout:(127/138): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-09-13T16:41:57.673 INFO:teuthology.orchestra.run.smithi191.stdout:(128/138): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-13T16:41:57.765 INFO:teuthology.orchestra.run.smithi191.stdout:(129/138): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-13T16:41:58.207 INFO:teuthology.orchestra.run.smithi191.stdout:(130/138): librados2-18.2.4-850.g50bbe7f3.el9.x 6.2 MB/s | 3.3 MB 00:00 2024-09-13T16:41:58.274 INFO:teuthology.orchestra.run.smithi191.stdout:(131/138): librbd1-18.2.4-850.g50bbe7f3.el9.x86 5.9 MB/s | 3.0 MB 00:00 2024-09-13T16:41:58.457 INFO:teuthology.orchestra.run.smithi191.stdout:(132/138): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.1 MB/s | 16 MB 00:02 2024-09-13T16:41:58.508 INFO:teuthology.orchestra.run.smithi191.stdout:(133/138): NetworkManager-team-1.49.90-1.el9.x8 819 kB/s | 41 kB 00:00 2024-09-13T16:41:58.584 INFO:teuthology.orchestra.run.smithi191.stdout:(134/138): NetworkManager-tui-1.49.90-1.el9.x86 3.2 MB/s | 249 kB 00:00 2024-09-13T16:41:58.708 INFO:teuthology.orchestra.run.smithi191.stdout:(135/138): openssl-3.2.2-6.el9.x86_64.rpm 11 MB/s | 1.4 MB 00:00 2024-09-13T16:41:58.750 INFO:teuthology.orchestra.run.smithi191.stdout:(136/138): NetworkManager-libnm-1.49.90-1.el9.x 3.8 MB/s | 1.8 MB 00:00 2024-09-13T16:41:58.800 INFO:teuthology.orchestra.run.smithi191.stdout:(137/138): NetworkManager-1.49.90-1.el9.x86_64. 3.9 MB/s | 2.3 MB 00:00 2024-09-13T16:41:58.934 INFO:teuthology.orchestra.run.smithi191.stdout:(138/138): openssl-libs-3.2.2-6.el9.x86_64.rpm 11 MB/s | 2.4 MB 00:00 2024-09-13T16:41:58.939 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-13T16:41:58.939 INFO:teuthology.orchestra.run.smithi191.stdout:Total 16 MB/s | 203 MB 00:12 2024-09-13T16:41:59.618 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-13T16:41:59.618 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-13T16:42:00.048 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-13T16:42:00.189 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-13T16:42:00.189 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-13T16:42:01.823 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-13T16:42:01.823 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-13T16:42:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-13T16:42:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/146 2024-09-13T16:42:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/146 2024-09-13T16:42:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/146 2024-09-13T16:42:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/146 2024-09-13T16:42:06.332 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/146 2024-09-13T16:42:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/146 2024-09-13T16:42:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/146 2024-09-13T16:42:06.890 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 8/146 2024-09-13T16:42:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 8/146 2024-09-13T16:42:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 9/146 2024-09-13T16:42:07.179 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 9/146 2024-09-13T16:42:07.212 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:18.2.4-850.g50bbe7f3.el9.x86_64 10/146 2024-09-13T16:42:07.428 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/146 2024-09-13T16:42:07.494 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/146 2024-09-13T16:42:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:18.2.4-850.g50bbe7f3.el9.x86_64 13/146 2024-09-13T16:42:07.585 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-13T16:42:07.665 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:18.2.4-850.g50bbe7f3.el9.x86_64 13/146 2024-09-13T16:42:07.689 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/146 2024-09-13T16:42:07.714 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:18.2.4-850.g50bbe7f3.el9.x86_ 14/146 2024-09-13T16:42:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/146 2024-09-13T16:42:07.781 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/146 2024-09-13T16:42:07.813 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/146 2024-09-13T16:42:07.838 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/146 2024-09-13T16:42:07.889 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/146 2024-09-13T16:42:07.978 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:18.2.4-850.g50bbe7f3.el9.x86_ 14/146 2024-09-13T16:42:08.183 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 15/146 2024-09-13T16:42:08.212 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/146 2024-09-13T16:42:08.318 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 16/146 2024-09-13T16:42:08.322 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 8/146 2024-09-13T16:42:08.442 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 8/146 2024-09-13T16:42:08.442 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 16/146 2024-09-13T16:42:08.474 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 9/146 2024-09-13T16:42:08.549 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 9/146 2024-09-13T16:42:08.578 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:18.2.4-850.g50bbe7f3.el9.x86_64 10/146 2024-09-13T16:42:08.727 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/146 2024-09-13T16:42:08.775 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/146 2024-09-13T16:42:08.811 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:18.2.4-850.g50bbe7f3.el9.x86_64 13/146 2024-09-13T16:42:08.814 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 16/146 2024-09-13T16:42:08.857 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 17/146 2024-09-13T16:42:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 18/146 2024-09-13T16:42:08.913 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:18.2.4-850.g50bbe7f3.el9.x86_64 13/146 2024-09-13T16:42:08.937 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:18.2.4-850.g50bbe7f3.el9.x86_ 14/146 2024-09-13T16:42:08.985 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/146 2024-09-13T16:42:09.034 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 20/146 2024-09-13T16:42:09.067 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 21/146 2024-09-13T16:42:09.084 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:18.2.4-850.g50bbe7f3.el9.x86_ 14/146 2024-09-13T16:42:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 22/146 2024-09-13T16:42:09.217 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 15/146 2024-09-13T16:42:09.221 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 23/146 2024-09-13T16:42:09.255 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 24/146 2024-09-13T16:42:09.319 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 25/146 2024-09-13T16:42:09.323 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 16/146 2024-09-13T16:42:09.393 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 16/146 2024-09-13T16:42:09.425 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 26/146 2024-09-13T16:42:09.458 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:18.2.4-850.g50bbe7f3.el9.x 27/146 2024-09-13T16:42:09.503 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:18.2.4-850.g50bbe7f3.el9 28/146 2024-09-13T16:42:09.593 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:18.2.4-850.g50bbe7f3.el9.x86_64 29/146 2024-09-13T16:42:09.676 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 16/146 2024-09-13T16:42:09.706 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 17/146 2024-09-13T16:42:09.741 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.7-2.el9.x86_64 18/146 2024-09-13T16:42:09.771 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 30/146 2024-09-13T16:42:09.815 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/146 2024-09-13T16:42:09.835 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 31/146 2024-09-13T16:42:09.856 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 20/146 2024-09-13T16:42:09.884 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 21/146 2024-09-13T16:42:09.970 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 22/146 2024-09-13T16:42:10.023 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 23/146 2024-09-13T16:42:10.053 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 24/146 2024-09-13T16:42:10.098 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 25/146 2024-09-13T16:42:10.174 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 26/146 2024-09-13T16:42:10.208 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:18.2.4-850.g50bbe7f3.el9.x 27/146 2024-09-13T16:42:10.251 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:18.2.4-850.g50bbe7f3.el9 28/146 2024-09-13T16:42:10.326 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:18.2.4-850.g50bbe7f3.el9.x86_64 29/146 2024-09-13T16:42:10.469 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 30/146 2024-09-13T16:42:10.507 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 32/146 2024-09-13T16:42:10.523 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 31/146 2024-09-13T16:42:10.629 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 33/146 2024-09-13T16:42:10.680 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 34/146 2024-09-13T16:42:10.718 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 35/146 2024-09-13T16:42:10.771 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 36/146 2024-09-13T16:42:11.005 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 37/146 2024-09-13T16:42:11.055 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-4.9-2.el9.noarch 32/146 2024-09-13T16:42:11.135 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 33/146 2024-09-13T16:42:11.174 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 34/146 2024-09-13T16:42:11.211 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 35/146 2024-09-13T16:42:11.253 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 36/146 2024-09-13T16:42:11.267 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 38/146 2024-09-13T16:42:11.413 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-20.el9.x86_64 39/146 2024-09-13T16:42:11.414 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 37/146 2024-09-13T16:42:11.478 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 38/146 2024-09-13T16:42:11.588 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : re2-1:20211101-20.el9.x86_64 39/146 2024-09-13T16:42:11.616 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 40/146 2024-09-13T16:42:11.633 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 40/146 2024-09-13T16:42:11.689 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 41/146 2024-09-13T16:42:11.691 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 41/146 2024-09-13T16:42:11.761 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 42/146 2024-09-13T16:42:11.778 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 42/146 2024-09-13T16:42:11.793 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 43/146 2024-09-13T16:42:11.822 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 43/146 2024-09-13T16:42:11.829 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 44/146 2024-09-13T16:42:11.858 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 44/146 2024-09-13T16:42:11.875 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 45/146 2024-09-13T16:42:11.918 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 45/146 2024-09-13T16:42:11.921 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 46/146 2024-09-13T16:42:11.963 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 47/146 2024-09-13T16:42:11.970 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 46/146 2024-09-13T16:42:12.021 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 47/146 2024-09-13T16:42:12.099 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 48/146 2024-09-13T16:42:12.148 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 49/146 2024-09-13T16:42:12.189 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 50/146 2024-09-13T16:42:12.201 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 48/146 2024-09-13T16:42:12.260 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 49/146 2024-09-13T16:42:12.293 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 51/146 2024-09-13T16:42:12.303 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 50/146 2024-09-13T16:42:12.332 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 52/146 2024-09-13T16:42:12.371 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 53/146 2024-09-13T16:42:12.397 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 54/146 2024-09-13T16:42:12.426 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 51/146 2024-09-13T16:42:12.433 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 55/146 2024-09-13T16:42:12.458 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fmt-8.1.1-5.el9.x86_64 56/146 2024-09-13T16:42:12.468 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 52/146 2024-09-13T16:42:12.486 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 57/146 2024-09-13T16:42:12.515 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 53/146 2024-09-13T16:42:12.520 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 58/146 2024-09-13T16:42:12.549 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 54/146 2024-09-13T16:42:12.557 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : uuid-1.6.2-55.el9.x86_64 59/146 2024-09-13T16:42:12.576 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 55/146 2024-09-13T16:42:12.595 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 60/146 2024-09-13T16:42:12.610 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fmt-8.1.1-5.el9.x86_64 56/146 2024-09-13T16:42:12.629 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 61/146 2024-09-13T16:42:12.639 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 57/146 2024-09-13T16:42:12.663 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 62/146 2024-09-13T16:42:12.675 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 58/146 2024-09-13T16:42:12.705 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 63/146 2024-09-13T16:42:12.719 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : uuid-1.6.2-55.el9.x86_64 59/146 2024-09-13T16:42:12.739 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 64/146 2024-09-13T16:42:12.756 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 60/146 2024-09-13T16:42:12.774 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 65/146 2024-09-13T16:42:12.790 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 61/146 2024-09-13T16:42:12.821 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 66/146 2024-09-13T16:42:12.832 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 62/146 2024-09-13T16:42:12.892 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 63/146 2024-09-13T16:42:12.920 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-packaging-20.9-5.el9.noarch 67/146 2024-09-13T16:42:12.945 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 64/146 2024-09-13T16:42:12.967 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 68/146 2024-09-13T16:42:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 65/146 2024-09-13T16:42:13.030 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 66/146 2024-09-13T16:42:13.159 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 67/146 2024-09-13T16:42:13.219 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 68/146 2024-09-13T16:42:13.975 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 69/146 2024-09-13T16:42:14.023 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 70/146 2024-09-13T16:42:14.056 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 71/146 2024-09-13T16:42:14.156 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 72/146 2024-09-13T16:42:14.167 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 69/146 2024-09-13T16:42:14.207 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 73/146 2024-09-13T16:42:14.228 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 70/146 2024-09-13T16:42:14.262 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 71/146 2024-09-13T16:42:14.379 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 72/146 2024-09-13T16:42:14.399 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.26-2.el9.x86_64 74/146 2024-09-13T16:42:14.420 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 73/146 2024-09-13T16:42:14.429 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 75/146 2024-09-13T16:42:14.508 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 76/146 2024-09-13T16:42:14.678 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.26-2.el9.x86_64 74/146 2024-09-13T16:42:14.708 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 75/146 2024-09-13T16:42:14.803 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 76/146 2024-09-13T16:42:15.147 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 77/146 2024-09-13T16:42:15.260 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 78/146 2024-09-13T16:42:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 77/146 2024-09-13T16:42:15.809 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 78/146 2024-09-13T16:42:16.479 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 79/146 2024-09-13T16:42:16.566 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 80/146 2024-09-13T16:42:16.594 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 81/146 2024-09-13T16:42:16.627 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 82/146 2024-09-13T16:42:16.926 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 83/146 2024-09-13T16:42:16.982 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 84/146 2024-09-13T16:42:17.073 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 84/146 2024-09-13T16:42:17.103 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 85/146 2024-09-13T16:42:17.139 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 86/146 2024-09-13T16:42:17.455 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 79/146 2024-09-13T16:42:17.545 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 87/146 2024-09-13T16:42:17.551 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 80/146 2024-09-13T16:42:17.584 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:18.2.4-850.g50bbe7f3.el9.x86_64 88/146 2024-09-13T16:42:17.595 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 81/146 2024-09-13T16:42:17.628 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 82/146 2024-09-13T16:42:17.655 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:18.2.4-850.g50bbe7f3.el9.x86_64 88/146 2024-09-13T16:42:17.678 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 89/146 2024-09-13T16:42:17.978 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 83/146 2024-09-13T16:42:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 84/146 2024-09-13T16:42:18.144 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 84/146 2024-09-13T16:42:18.179 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 85/146 2024-09-13T16:42:18.228 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 86/146 2024-09-13T16:42:18.694 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 87/146 2024-09-13T16:42:18.725 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:18.2.4-850.g50bbe7f3.el9.x86_64 88/146 2024-09-13T16:42:18.816 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:18.2.4-850.g50bbe7f3.el9.x86_64 88/146 2024-09-13T16:42:18.844 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 89/146 2024-09-13T16:42:19.754 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 90/146 2024-09-13T16:42:19.791 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 90/146 2024-09-13T16:42:20.313 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 90/146 2024-09-13T16:42:20.352 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 91/146 2024-09-13T16:42:20.453 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 91/146 2024-09-13T16:42:20.454 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T16:42:20.454 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T16:42:20.454 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:42:20.487 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:18.2.4-850.g50bbe7f3.el9.x86_64 92/146 2024-09-13T16:42:21.531 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 90/146 2024-09-13T16:42:21.573 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 90/146 2024-09-13T16:42:22.209 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 90/146 2024-09-13T16:42:22.251 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 91/146 2024-09-13T16:42:22.421 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 91/146 2024-09-13T16:42:22.421 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T16:42:22.421 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T16:42:22.421 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:42:22.494 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:18.2.4-850.g50bbe7f3.el9.x86_64 92/146 2024-09-13T16:42:32.130 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:18.2.4-850.g50bbe7f3.el9.x86_64 92/146 2024-09-13T16:42:32.130 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2024-09-13T16:42:32.130 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2024-09-13T16:42:32.131 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2024-09-13T16:42:32.131 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2024-09-13T16:42:32.131 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2024-09-13T16:42:32.131 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /root 2024-09-13T16:42:32.131 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2024-09-13T16:42:32.131 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:42:32.305 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-volume-2:18.2.4-850.g50bbe7f3.el9.noarch 93/146 2024-09-13T16:42:32.322 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-volume-2:18.2.4-850.g50bbe7f3.el9.noarch 93/146 2024-09-13T16:42:32.322 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:32.322 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T16:42:32.322 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:42:33.783 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:18.2.4-850.g50bbe7f3.el9.x86_64 94/146 2024-09-13T16:42:33.809 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:18.2.4-850.g50bbe7f3.el9.x86_64 94/146 2024-09-13T16:42:33.809 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:33.810 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T16:42:33.810 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T16:42:33.810 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T16:42:33.810 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:42:34.008 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:18.2.4-850.g50bbe7f3.el9.x86_64 95/146 2024-09-13T16:42:34.031 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:18.2.4-850.g50bbe7f3.el9.x86_64 95/146 2024-09-13T16:42:34.031 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:34.031 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T16:42:34.032 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T16:42:34.032 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T16:42:34.032 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:42:34.467 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:18.2.4-850.g50bbe7f3.el9.x86_64 96/146 2024-09-13T16:42:34.491 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:18.2.4-850.g50bbe7f3.el9.x86_64 96/146 2024-09-13T16:42:34.491 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:34.491 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T16:42:34.491 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T16:42:34.491 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T16:42:34.491 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:42:34.589 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : automake-1.16.2-8.el9.noarch 97/146 2024-09-13T16:42:34.637 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ply-3.11-14.el9.noarch 98/146 2024-09-13T16:42:34.692 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 99/146 2024-09-13T16:42:34.875 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 100/146 2024-09-13T16:42:34.926 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 101/146 2024-09-13T16:42:34.994 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 102/146 2024-09-13T16:42:35.091 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 103/146 2024-09-13T16:42:35.229 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/146 2024-09-13T16:42:35.268 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 105/146 2024-09-13T16:42:35.299 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 106/146 2024-09-13T16:42:35.327 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.9-16.el9.x86_64 107/146 2024-09-13T16:42:35.463 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:18.2.4-850.g50bbe7f3.el9.x86_64 92/146 2024-09-13T16:42:35.463 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-09-13T16:42:35.463 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-09-13T16:42:35.463 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-09-13T16:42:35.463 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-09-13T16:42:35.464 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-09-13T16:42:35.464 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-09-13T16:42:35.464 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-09-13T16:42:35.464 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:42:35.669 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:18.2.4-850.g50bbe7f3.el9.noarch 93/146 2024-09-13T16:42:35.706 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:18.2.4-850.g50bbe7f3.el9.noarch 93/146 2024-09-13T16:42:35.706 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:35.706 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T16:42:35.706 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:42:35.716 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 108/146 2024-09-13T16:42:35.742 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 108/146 2024-09-13T16:42:35.771 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-850.g50bbe7f3.el 109/146 2024-09-13T16:42:35.833 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-850.g50bbe7f3.e 110/146 2024-09-13T16:42:35.857 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-850.g50bbe7f3.el9.noar 111/146 2024-09-13T16:42:36.887 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-850.g50bbe7f3.el9.noar 111/146 2024-09-13T16:42:36.917 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-850.g50bbe7f3.el9.no 112/146 2024-09-13T16:42:37.323 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:18.2.4-850.g50bbe7f3.el9.x86_64 94/146 2024-09-13T16:42:37.382 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:18.2.4-850.g50bbe7f3.el9.x86_64 94/146 2024-09-13T16:42:37.382 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:37.382 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T16:42:37.382 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T16:42:37.382 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T16:42:37.382 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:42:37.615 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:18.2.4-850.g50bbe7f3.el9.x86_64 95/146 2024-09-13T16:42:37.666 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:18.2.4-850.g50bbe7f3.el9.x86_64 95/146 2024-09-13T16:42:37.667 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:37.667 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T16:42:37.667 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T16:42:37.667 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T16:42:37.667 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:42:37.872 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-850.g50bbe7f3.el9.no 112/146 2024-09-13T16:42:37.893 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-850.g50bb 113/146 2024-09-13T16:42:37.996 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-850.g50bb 113/146 2024-09-13T16:42:38.156 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-850.g50bbe7f3.el9 114/146 2024-09-13T16:42:38.174 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:18.2.4-850.g50bbe7f3.el9.x86_64 96/146 2024-09-13T16:42:38.186 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 115/146 2024-09-13T16:42:38.210 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 115/146 2024-09-13T16:42:38.210 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:38.210 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T16:42:38.210 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T16:42:38.210 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T16:42:38.211 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:42:38.226 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:18.2.4-850.g50bbe7f3.el9.x86_64 96/146 2024-09-13T16:42:38.226 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:38.226 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T16:42:38.226 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T16:42:38.226 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T16:42:38.226 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:42:38.262 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:18.2.4-850.g50bbe7f3.el9.noarch 116/146 2024-09-13T16:42:38.274 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-850.g50bbe7f3.el9.noarch 116/146 2024-09-13T16:42:38.358 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : automake-1.16.2-8.el9.noarch 97/146 2024-09-13T16:42:38.368 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:18.2.4-850.g50bbe7f3.el9.x86_64 117/146 2024-09-13T16:42:38.418 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 118/146 2024-09-13T16:42:38.419 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 98/146 2024-09-13T16:42:38.510 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 99/146 2024-09-13T16:42:38.751 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 100/146 2024-09-13T16:42:38.812 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 101/146 2024-09-13T16:42:38.902 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 102/146 2024-09-13T16:42:39.010 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 103/146 2024-09-13T16:42:39.131 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libtool-2.4.6-46.el9.x86_64 119/146 2024-09-13T16:42:39.156 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 120/146 2024-09-13T16:42:39.179 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 120/146 2024-09-13T16:42:39.179 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:39.179 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T16:42:39.179 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T16:42:39.179 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T16:42:39.179 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:42:39.182 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/146 2024-09-13T16:42:39.223 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-850.g50bbe7 121/146 2024-09-13T16:42:39.236 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 105/146 2024-09-13T16:42:39.247 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-850.g50bbe7 121/146 2024-09-13T16:42:39.247 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:39.247 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T16:42:39.247 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:42:39.276 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 106/146 2024-09-13T16:42:39.290 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 122/146 2024-09-13T16:42:39.309 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-16.el9.x86_64 107/146 2024-09-13T16:42:39.314 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephfs-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 122/146 2024-09-13T16:42:39.314 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:39.314 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-13T16:42:39.314 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-13T16:42:39.315 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-13T16:42:39.315 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:42:39.616 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 123/146 2024-09-13T16:42:39.640 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 123/146 2024-09-13T16:42:39.640 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:39.640 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T16:42:39.640 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T16:42:39.640 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T16:42:39.640 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:42:39.852 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 108/146 2024-09-13T16:42:39.892 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 108/146 2024-09-13T16:42:39.930 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-850.g50bbe7f3.el 109/146 2024-09-13T16:42:40.002 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-850.g50bbe7f3.e 110/146 2024-09-13T16:42:40.030 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-850.g50bbe7f3.el9.noar 111/146 2024-09-13T16:42:41.320 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-850.g50bbe7f3.el9.noar 111/146 2024-09-13T16:42:41.357 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-850.g50bbe7f3.el9.no 112/146 2024-09-13T16:42:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-850.g50bbe7f3.el9.no 112/146 2024-09-13T16:42:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-850.g50bb 113/146 2024-09-13T16:42:42.692 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-850.g50bb 113/146 2024-09-13T16:42:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-850.g50bbe7f3.el9 114/146 2024-09-13T16:42:42.892 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 115/146 2024-09-13T16:42:42.945 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 115/146 2024-09-13T16:42:42.945 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:42.945 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T16:42:42.945 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T16:42:42.945 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T16:42:42.945 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:42:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:18.2.4-850.g50bbe7f3.el9.noarch 116/146 2024-09-13T16:42:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-850.g50bbe7f3.el9.noarch 116/146 2024-09-13T16:42:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:18.2.4-850.g50bbe7f3.el9.x86_64 117/146 2024-09-13T16:42:43.210 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 118/146 2024-09-13T16:42:43.312 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:18.2.4-850.g50bbe7f3.el9.x86_64 124/146 2024-09-13T16:42:43.353 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 125/146 2024-09-13T16:42:43.394 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 126/146 2024-09-13T16:42:43.425 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 127/146 2024-09-13T16:42:43.600 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 128/146 2024-09-13T16:42:43.649 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 129/146 2024-09-13T16:42:43.680 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:18.2.4-850.g50bbe7f3.el9.x86_6 130/146 2024-09-13T16:42:43.709 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 131/146 2024-09-13T16:42:43.755 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 132/146 2024-09-13T16:42:44.043 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libtool-2.4.6-46.el9.x86_64 119/146 2024-09-13T16:42:44.079 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 120/146 2024-09-13T16:42:44.131 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 120/146 2024-09-13T16:42:44.132 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:44.132 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T16:42:44.132 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T16:42:44.132 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T16:42:44.132 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:42:44.179 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-850.g50bbe7 121/146 2024-09-13T16:42:44.229 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-850.g50bbe7 121/146 2024-09-13T16:42:44.229 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:44.229 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T16:42:44.229 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:42:44.288 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 122/146 2024-09-13T16:42:44.339 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephfs-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 122/146 2024-09-13T16:42:44.340 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:44.340 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-13T16:42:44.340 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-13T16:42:44.340 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-13T16:42:44.340 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:42:44.355 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 133/146 2024-09-13T16:42:44.394 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 133/146 2024-09-13T16:42:44.415 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 133/146 2024-09-13T16:42:44.731 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 123/146 2024-09-13T16:42:44.783 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 123/146 2024-09-13T16:42:44.783 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T16:42:44.783 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T16:42:44.783 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T16:42:44.783 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T16:42:44.783 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:42:45.247 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-top-2:18.2.4-850.g50bbe7f3.el9.noarch 134/146 2024-09-13T16:42:45.341 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 135/146 2024-09-13T16:42:45.382 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 136/146 2024-09-13T16:42:45.463 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flex-2.6.4-9.el9.x86_64 137/146 2024-09-13T16:42:45.494 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : bison-3.7.4-5.el9.x86_64 138/146 2024-09-13T16:42:45.519 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 139/146 2024-09-13T16:42:45.545 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 140/146 2024-09-13T16:42:45.545 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 141/146 2024-09-13T16:42:45.629 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 141/146 2024-09-13T16:42:45.629 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 142/146 2024-09-13T16:42:45.713 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 142/146 2024-09-13T16:42:45.738 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 143/146 2024-09-13T16:42:45.743 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 144/146 2024-09-13T16:42:45.744 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 144/146 2024-09-13T16:42:45.923 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 144/146 2024-09-13T16:42:45.923 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 145/146 2024-09-13T16:42:46.007 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 145/146 2024-09-13T16:42:46.070 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 146/146 2024-09-13T16:42:48.995 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:18.2.4-850.g50bbe7f3.el9.x86_64 124/146 2024-09-13T16:42:49.044 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 125/146 2024-09-13T16:42:49.088 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 126/146 2024-09-13T16:42:49.124 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 127/146 2024-09-13T16:42:49.368 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 128/146 2024-09-13T16:42:49.422 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 129/146 2024-09-13T16:42:49.452 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:18.2.4-850.g50bbe7f3.el9.x86_6 130/146 2024-09-13T16:42:49.484 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 131/146 2024-09-13T16:42:49.527 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 132/146 2024-09-13T16:42:50.209 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 133/146 2024-09-13T16:42:50.258 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 133/146 2024-09-13T16:42:50.301 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 133/146 2024-09-13T16:42:51.674 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 146/146 2024-09-13T16:42:51.674 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/146 2024-09-13T16:42:51.674 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 2/146 2024-09-13T16:42:51.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 3/146 2024-09-13T16:42:51.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 4/146 2024-09-13T16:42:51.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-850.g50bbe7 5/146 2024-09-13T16:42:51.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:18.2.4-850.g50bbe7f3.el9.x86_64 6/146 2024-09-13T16:42:51.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 7/146 2024-09-13T16:42:51.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:18.2.4-850.g50bbe7f3.el9.x86_64 8/146 2024-09-13T16:42:51.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:18.2.4-850.g50bbe7f3.el9.x86_64 9/146 2024-09-13T16:42:51.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 10/146 2024-09-13T16:42:51.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:18.2.4-850.g50bbe7f3.el9.x86_64 11/146 2024-09-13T16:42:51.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:18.2.4-850.g50bbe7f3.el9.x86_64 12/146 2024-09-13T16:42:51.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 13/146 2024-09-13T16:42:51.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:18.2.4-850.g50bbe7f3.el9.x86_6 14/146 2024-09-13T16:42:51.675 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 15/146 2024-09-13T16:42:51.676 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:18.2.4-850.g50bbe7f3.el9.x86_64 16/146 2024-09-13T16:42:51.676 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 17/146 2024-09-13T16:42:51.676 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:18.2.4-850.g50bbe7f3.el9.x86_ 18/146 2024-09-13T16:42:51.676 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:18.2.4-850.g50bbe7f3.el9.x86_64 19/146 2024-09-13T16:42:51.676 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:18.2.4-850.g50bbe7f3.el9 20/146 2024-09-13T16:42:51.676 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:18.2.4-850.g50bbe7f3.el9.x 21/146 2024-09-13T16:42:51.676 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:18.2.4-850.g50bbe7f3.el9.x86_64 22/146 2024-09-13T16:42:51.676 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:18.2.4-850.g50bbe7f3.el9.x86_64 23/146 2024-09-13T16:42:51.676 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 24/146 2024-09-13T16:42:51.676 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 25/146 2024-09-13T16:42:51.676 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 26/146 2024-09-13T16:42:51.676 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 27/146 2024-09-13T16:42:51.676 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 28/146 2024-09-13T16:42:51.677 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-850.g50bbe7f3.e 29/146 2024-09-13T16:42:51.677 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-850.g50bbe7f3.el9.noar 30/146 2024-09-13T16:42:51.677 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-top-2:18.2.4-850.g50bbe7f3.el9.noarch 134/146 2024-09-13T16:42:51.687 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-850.g50bbe7f3.el9.no 31/146 2024-09-13T16:42:51.688 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-850.g50bb 32/146 2024-09-13T16:42:51.688 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-850.g50bbe7f3.el9 33/146 2024-09-13T16:42:51.688 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:18.2.4-850.g50bbe7f3.el9.noarch 34/146 2024-09-13T16:42:51.688 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-850.g50bbe7f3.el 35/146 2024-09-13T16:42:51.688 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-volume-2:18.2.4-850.g50bbe7f3.el9.noarch 36/146 2024-09-13T16:42:51.688 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 37/146 2024-09-13T16:42:51.688 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-top-2:18.2.4-850.g50bbe7f3.el9.noarch 38/146 2024-09-13T16:42:51.688 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 39/146 2024-09-13T16:42:51.688 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 40/146 2024-09-13T16:42:51.688 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 41/146 2024-09-13T16:42:51.688 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 42/146 2024-09-13T16:42:51.688 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 43/146 2024-09-13T16:42:51.688 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ply-3.11-14.el9.noarch 44/146 2024-09-13T16:42:51.689 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 45/146 2024-09-13T16:42:51.689 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 46/146 2024-09-13T16:42:51.689 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 47/146 2024-09-13T16:42:51.689 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : automake-1.16.2-8.el9.noarch 48/146 2024-09-13T16:42:51.689 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : bison-3.7.4-5.el9.x86_64 49/146 2024-09-13T16:42:51.689 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 50/146 2024-09-13T16:42:51.689 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 51/146 2024-09-13T16:42:51.689 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flex-2.6.4-9.el9.x86_64 52/146 2024-09-13T16:42:51.689 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 53/146 2024-09-13T16:42:51.689 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 54/146 2024-09-13T16:42:51.689 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 55/146 2024-09-13T16:42:51.689 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 56/146 2024-09-13T16:42:51.689 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 57/146 2024-09-13T16:42:51.690 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 58/146 2024-09-13T16:42:51.690 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 59/146 2024-09-13T16:42:51.690 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 60/146 2024-09-13T16:42:51.690 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 61/146 2024-09-13T16:42:51.690 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 62/146 2024-09-13T16:42:51.690 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 63/146 2024-09-13T16:42:51.690 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 64/146 2024-09-13T16:42:51.690 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 65/146 2024-09-13T16:42:51.690 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 66/146 2024-09-13T16:42:51.690 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-13.16-1.el9.x86_64 67/146 2024-09-13T16:42:51.690 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 68/146 2024-09-13T16:42:51.690 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 69/146 2024-09-13T16:42:51.690 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 70/146 2024-09-13T16:42:51.691 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 71/146 2024-09-13T16:42:51.691 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 72/146 2024-09-13T16:42:51.691 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 73/146 2024-09-13T16:42:51.691 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 74/146 2024-09-13T16:42:51.691 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 75/146 2024-09-13T16:42:51.691 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 76/146 2024-09-13T16:42:51.691 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 77/146 2024-09-13T16:42:51.691 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 78/146 2024-09-13T16:42:51.691 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 79/146 2024-09-13T16:42:51.691 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 80/146 2024-09-13T16:42:51.691 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 81/146 2024-09-13T16:42:51.691 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 82/146 2024-09-13T16:42:51.691 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 83/146 2024-09-13T16:42:51.692 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 84/146 2024-09-13T16:42:51.692 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 85/146 2024-09-13T16:42:51.692 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/146 2024-09-13T16:42:51.692 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 87/146 2024-09-13T16:42:51.692 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 88/146 2024-09-13T16:42:51.692 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 89/146 2024-09-13T16:42:51.692 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 90/146 2024-09-13T16:42:51.692 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 91/146 2024-09-13T16:42:51.692 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 92/146 2024-09-13T16:42:51.692 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 93/146 2024-09-13T16:42:51.692 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 94/146 2024-09-13T16:42:51.692 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 95/146 2024-09-13T16:42:51.692 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 96/146 2024-09-13T16:42:51.693 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 97/146 2024-09-13T16:42:51.693 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 98/146 2024-09-13T16:42:51.693 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 99/146 2024-09-13T16:42:51.693 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 100/146 2024-09-13T16:42:51.693 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 101/146 2024-09-13T16:42:51.693 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 102/146 2024-09-13T16:42:51.693 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 103/146 2024-09-13T16:42:51.693 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 104/146 2024-09-13T16:42:51.693 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 105/146 2024-09-13T16:42:51.693 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 106/146 2024-09-13T16:42:51.693 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 107/146 2024-09-13T16:42:51.693 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 108/146 2024-09-13T16:42:51.693 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 109/146 2024-09-13T16:42:51.693 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 110/146 2024-09-13T16:42:51.694 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 111/146 2024-09-13T16:42:51.694 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 112/146 2024-09-13T16:42:51.694 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 113/146 2024-09-13T16:42:51.694 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 114/146 2024-09-13T16:42:51.694 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 115/146 2024-09-13T16:42:51.694 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 116/146 2024-09-13T16:42:51.694 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 117/146 2024-09-13T16:42:51.694 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 118/146 2024-09-13T16:42:51.694 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/146 2024-09-13T16:42:51.694 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 120/146 2024-09-13T16:42:51.694 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 121/146 2024-09-13T16:42:51.694 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 122/146 2024-09-13T16:42:51.694 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 123/146 2024-09-13T16:42:51.695 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 124/146 2024-09-13T16:42:51.695 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 125/146 2024-09-13T16:42:51.695 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 126/146 2024-09-13T16:42:51.695 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 127/146 2024-09-13T16:42:51.695 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 128/146 2024-09-13T16:42:51.695 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : re2-1:20211101-20.el9.x86_64 129/146 2024-09-13T16:42:51.695 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 130/146 2024-09-13T16:42:51.695 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 131/146 2024-09-13T16:42:51.695 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 132/146 2024-09-13T16:42:51.695 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 133/146 2024-09-13T16:42:51.695 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 134/146 2024-09-13T16:42:51.695 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 135/146 2024-09-13T16:42:51.695 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 136/146 2024-09-13T16:42:51.695 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 137/146 2024-09-13T16:42:51.696 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 138/146 2024-09-13T16:42:51.696 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 139/146 2024-09-13T16:42:51.696 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 140/146 2024-09-13T16:42:51.696 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 141/146 2024-09-13T16:42:51.696 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 142/146 2024-09-13T16:42:51.696 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 143/146 2024-09-13T16:42:51.696 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 144/146 2024-09-13T16:42:51.696 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 145/146 2024-09-13T16:42:51.789 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 135/146 2024-09-13T16:42:51.846 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 136/146 2024-09-13T16:42:51.963 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flex-2.6.4-9.el9.x86_64 137/146 2024-09-13T16:42:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : bison-3.7.4-5.el9.x86_64 138/146 2024-09-13T16:42:52.032 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 139/146 2024-09-13T16:42:52.077 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 140/146 2024-09-13T16:42:52.077 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 141/146 2024-09-13T16:42:52.177 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 141/146 2024-09-13T16:42:52.177 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 142/146 2024-09-13T16:42:52.269 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 142/146 2024-09-13T16:42:52.293 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 143/146 2024-09-13T16:42:52.309 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 144/146 2024-09-13T16:42:52.309 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 144/146 2024-09-13T16:42:52.579 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 144/146 2024-09-13T16:42:52.579 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 145/146 2024-09-13T16:42:52.680 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 145/146 2024-09-13T16:42:52.756 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 146/146 2024-09-13T16:42:52.959 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 146/146 2024-09-13T16:42:52.959 INFO:teuthology.orchestra.run.smithi028.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-09-13T16:42:52.959 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:42:54.185 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 146/146 2024-09-13T16:42:54.185 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:42:54.185 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2024-09-13T16:42:54.186 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-13T16:42:54.186 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-13T16:42:54.186 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-13T16:42:54.186 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-13T16:42:54.186 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.186 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.186 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-13T16:42:54.186 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-13T16:42:54.186 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-13T16:42:54.187 INFO:teuthology.orchestra.run.smithi191.stdout: automake-1.16.2-8.el9.noarch 2024-09-13T16:42:54.187 INFO:teuthology.orchestra.run.smithi191.stdout: bison-3.7.4-5.el9.x86_64 2024-09-13T16:42:54.187 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T16:42:54.187 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.187 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.187 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.187 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.187 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:42:54.187 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:42:54.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:42:54.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:42:54.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:42:54.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:42:54.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:42:54.188 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.189 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.189 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.189 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:42:54.189 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:42:54.189 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.189 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:42:54.189 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-13T16:42:54.189 INFO:teuthology.orchestra.run.smithi191.stdout: flex-2.6.4-9.el9.x86_64 2024-09-13T16:42:54.189 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T16:42:54.190 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T16:42:54.190 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T16:42:54.190 INFO:teuthology.orchestra.run.smithi191.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-13T16:42:54.190 INFO:teuthology.orchestra.run.smithi191.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-13T16:42:54.190 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T16:42:54.190 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-13T16:42:54.190 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T16:42:54.190 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T16:42:54.190 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.190 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.191 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.191 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T16:42:54.191 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T16:42:54.191 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T16:42:54.191 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T16:42:54.191 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T16:42:54.191 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.191 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.191 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T16:42:54.191 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.192 INFO:teuthology.orchestra.run.smithi191.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-13T16:42:54.192 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T16:42:54.192 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-13T16:42:54.192 INFO:teuthology.orchestra.run.smithi191.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-13T16:42:54.192 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T16:42:54.192 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T16:42:54.193 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T16:42:54.193 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-13T16:42:54.193 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T16:42:54.193 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-13.16-1.el9.x86_64 2024-09-13T16:42:54.193 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-13T16:42:54.193 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-13T16:42:54.193 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-13T16:42:54.193 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T16:42:54.193 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T16:42:54.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T16:42:54.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T16:42:54.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T16:42:54.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T16:42:54.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T16:42:54.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T16:42:54.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T16:42:54.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T16:42:54.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T16:42:54.194 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T16:42:54.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T16:42:54.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T16:42:54.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T16:42:54.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T16:42:54.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T16:42:54.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T16:42:54.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T16:42:54.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T16:42:54.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T16:42:54.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T16:42:54.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T16:42:54.195 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T16:42:54.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T16:42:54.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T16:42:54.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T16:42:54.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T16:42:54.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T16:42:54.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T16:42:54.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T16:42:54.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T16:42:54.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T16:42:54.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T16:42:54.196 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T16:42:54.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T16:42:54.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T16:42:54.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T16:42:54.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T16:42:54.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T16:42:54.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T16:42:54.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T16:42:54.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T16:42:54.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T16:42:54.197 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T16:42:54.198 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T16:42:54.198 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T16:42:54.198 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T16:42:54.198 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T16:42:54.198 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T16:42:54.198 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T16:42:54.198 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T16:42:54.198 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.198 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.198 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:42:54.198 INFO:teuthology.orchestra.run.smithi191.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T16:42:54.198 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T16:42:54.198 INFO:teuthology.orchestra.run.smithi191.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T16:42:54.199 INFO:teuthology.orchestra.run.smithi191.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-13T16:42:54.199 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-13T16:42:54.199 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T16:42:54.199 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-13T16:42:54.199 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:42:54.199 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-13T16:42:54.578 DEBUG:teuthology.parallel:result is None 2024-09-13T16:42:58.246 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/146 2024-09-13T16:42:58.246 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 2/146 2024-09-13T16:42:58.246 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 3/146 2024-09-13T16:42:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 4/146 2024-09-13T16:42:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-850.g50bbe7 5/146 2024-09-13T16:42:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:18.2.4-850.g50bbe7f3.el9.x86_64 6/146 2024-09-13T16:42:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 7/146 2024-09-13T16:42:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:18.2.4-850.g50bbe7f3.el9.x86_64 8/146 2024-09-13T16:42:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:18.2.4-850.g50bbe7f3.el9.x86_64 9/146 2024-09-13T16:42:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 10/146 2024-09-13T16:42:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:18.2.4-850.g50bbe7f3.el9.x86_64 11/146 2024-09-13T16:42:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:18.2.4-850.g50bbe7f3.el9.x86_64 12/146 2024-09-13T16:42:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 13/146 2024-09-13T16:42:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:18.2.4-850.g50bbe7f3.el9.x86_6 14/146 2024-09-13T16:42:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 15/146 2024-09-13T16:42:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:18.2.4-850.g50bbe7f3.el9.x86_64 16/146 2024-09-13T16:42:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 17/146 2024-09-13T16:42:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:18.2.4-850.g50bbe7f3.el9.x86_ 18/146 2024-09-13T16:42:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:18.2.4-850.g50bbe7f3.el9.x86_64 19/146 2024-09-13T16:42:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:18.2.4-850.g50bbe7f3.el9 20/146 2024-09-13T16:42:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:18.2.4-850.g50bbe7f3.el9.x 21/146 2024-09-13T16:42:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:18.2.4-850.g50bbe7f3.el9.x86_64 22/146 2024-09-13T16:42:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:18.2.4-850.g50bbe7f3.el9.x86_64 23/146 2024-09-13T16:42:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 24/146 2024-09-13T16:42:58.249 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 25/146 2024-09-13T16:42:58.249 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 26/146 2024-09-13T16:42:58.249 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 27/146 2024-09-13T16:42:58.260 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 28/146 2024-09-13T16:42:58.260 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-850.g50bbe7f3.e 29/146 2024-09-13T16:42:58.260 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-850.g50bbe7f3.el9.noar 30/146 2024-09-13T16:42:58.261 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-850.g50bbe7f3.el9.no 31/146 2024-09-13T16:42:58.261 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-850.g50bb 32/146 2024-09-13T16:42:58.261 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-850.g50bbe7f3.el9 33/146 2024-09-13T16:42:58.261 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:18.2.4-850.g50bbe7f3.el9.noarch 34/146 2024-09-13T16:42:58.261 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-850.g50bbe7f3.el 35/146 2024-09-13T16:42:58.261 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:18.2.4-850.g50bbe7f3.el9.noarch 36/146 2024-09-13T16:42:58.261 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 37/146 2024-09-13T16:42:58.261 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-top-2:18.2.4-850.g50bbe7f3.el9.noarch 38/146 2024-09-13T16:42:58.261 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 39/146 2024-09-13T16:42:58.261 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 40/146 2024-09-13T16:42:58.262 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 41/146 2024-09-13T16:42:58.262 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 42/146 2024-09-13T16:42:58.262 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 43/146 2024-09-13T16:42:58.262 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 44/146 2024-09-13T16:42:58.262 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 45/146 2024-09-13T16:42:58.262 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 46/146 2024-09-13T16:42:58.262 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 47/146 2024-09-13T16:42:58.262 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : automake-1.16.2-8.el9.noarch 48/146 2024-09-13T16:42:58.262 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : bison-3.7.4-5.el9.x86_64 49/146 2024-09-13T16:42:58.262 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 50/146 2024-09-13T16:42:58.262 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 51/146 2024-09-13T16:42:58.263 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flex-2.6.4-9.el9.x86_64 52/146 2024-09-13T16:42:58.263 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 53/146 2024-09-13T16:42:58.263 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 54/146 2024-09-13T16:42:58.263 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 55/146 2024-09-13T16:42:58.263 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 56/146 2024-09-13T16:42:58.263 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 57/146 2024-09-13T16:42:58.263 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 58/146 2024-09-13T16:42:58.263 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 59/146 2024-09-13T16:42:58.263 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 60/146 2024-09-13T16:42:58.263 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 61/146 2024-09-13T16:42:58.264 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 62/146 2024-09-13T16:42:58.264 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 63/146 2024-09-13T16:42:58.264 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 64/146 2024-09-13T16:42:58.264 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 65/146 2024-09-13T16:42:58.264 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 66/146 2024-09-13T16:42:58.264 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-13.16-1.el9.x86_64 67/146 2024-09-13T16:42:58.264 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 68/146 2024-09-13T16:42:58.265 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 69/146 2024-09-13T16:42:58.265 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 70/146 2024-09-13T16:42:58.265 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 71/146 2024-09-13T16:42:58.265 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 72/146 2024-09-13T16:42:58.265 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 73/146 2024-09-13T16:42:58.265 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 74/146 2024-09-13T16:42:58.266 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 75/146 2024-09-13T16:42:58.266 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 76/146 2024-09-13T16:42:58.266 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 77/146 2024-09-13T16:42:58.266 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 78/146 2024-09-13T16:42:58.266 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 79/146 2024-09-13T16:42:58.266 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 80/146 2024-09-13T16:42:58.266 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 81/146 2024-09-13T16:42:58.266 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 82/146 2024-09-13T16:42:58.267 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 83/146 2024-09-13T16:42:58.267 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 84/146 2024-09-13T16:42:58.267 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 85/146 2024-09-13T16:42:58.267 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/146 2024-09-13T16:42:58.267 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 87/146 2024-09-13T16:42:58.267 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 88/146 2024-09-13T16:42:58.267 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 89/146 2024-09-13T16:42:58.267 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 90/146 2024-09-13T16:42:58.267 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 91/146 2024-09-13T16:42:58.267 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 92/146 2024-09-13T16:42:58.269 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 93/146 2024-09-13T16:42:58.269 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 94/146 2024-09-13T16:42:58.269 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 95/146 2024-09-13T16:42:58.269 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 96/146 2024-09-13T16:42:58.269 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 97/146 2024-09-13T16:42:58.269 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 98/146 2024-09-13T16:42:58.269 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 99/146 2024-09-13T16:42:58.270 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 100/146 2024-09-13T16:42:58.270 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 101/146 2024-09-13T16:42:58.270 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 102/146 2024-09-13T16:42:58.270 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 103/146 2024-09-13T16:42:58.270 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 104/146 2024-09-13T16:42:58.270 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 105/146 2024-09-13T16:42:58.270 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 106/146 2024-09-13T16:42:58.270 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 107/146 2024-09-13T16:42:58.270 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 108/146 2024-09-13T16:42:58.271 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 109/146 2024-09-13T16:42:58.271 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 110/146 2024-09-13T16:42:58.271 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 111/146 2024-09-13T16:42:58.271 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 112/146 2024-09-13T16:42:58.271 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 113/146 2024-09-13T16:42:58.271 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 114/146 2024-09-13T16:42:58.271 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 115/146 2024-09-13T16:42:58.271 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 116/146 2024-09-13T16:42:58.271 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 117/146 2024-09-13T16:42:58.271 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 118/146 2024-09-13T16:42:58.271 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/146 2024-09-13T16:42:58.271 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 120/146 2024-09-13T16:42:58.271 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 121/146 2024-09-13T16:42:58.272 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 122/146 2024-09-13T16:42:58.272 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 123/146 2024-09-13T16:42:58.272 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 124/146 2024-09-13T16:42:58.272 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 125/146 2024-09-13T16:42:58.272 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 126/146 2024-09-13T16:42:58.272 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 127/146 2024-09-13T16:42:58.272 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 128/146 2024-09-13T16:42:58.272 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-20.el9.x86_64 129/146 2024-09-13T16:42:58.272 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 130/146 2024-09-13T16:42:58.272 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 131/146 2024-09-13T16:42:58.272 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 132/146 2024-09-13T16:42:58.272 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 133/146 2024-09-13T16:42:58.273 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 134/146 2024-09-13T16:42:58.273 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 135/146 2024-09-13T16:42:58.273 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 136/146 2024-09-13T16:42:58.273 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 137/146 2024-09-13T16:42:58.273 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 138/146 2024-09-13T16:42:58.273 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 139/146 2024-09-13T16:42:58.273 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 140/146 2024-09-13T16:42:58.273 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 141/146 2024-09-13T16:42:58.273 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 142/146 2024-09-13T16:42:58.273 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 143/146 2024-09-13T16:42:58.273 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 144/146 2024-09-13T16:42:58.273 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 145/146 2024-09-13T16:43:00.438 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 146/146 2024-09-13T16:43:00.438 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:00.438 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-09-13T16:43:00.438 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-13T16:43:00.438 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-13T16:43:00.438 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-13T16:43:00.439 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-13T16:43:00.439 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.439 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.439 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-13T16:43:00.439 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-13T16:43:00.439 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-13T16:43:00.439 INFO:teuthology.orchestra.run.smithi028.stdout: automake-1.16.2-8.el9.noarch 2024-09-13T16:43:00.439 INFO:teuthology.orchestra.run.smithi028.stdout: bison-3.7.4-5.el9.x86_64 2024-09-13T16:43:00.439 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T16:43:00.439 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.439 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.439 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.439 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.440 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:43:00.440 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.440 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.440 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.440 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:43:00.440 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:43:00.440 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:43:00.440 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:43:00.440 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:43:00.440 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.440 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.440 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:43:00.441 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.441 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.441 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.441 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:43:00.441 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:43:00.441 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.441 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T16:43:00.441 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-13T16:43:00.441 INFO:teuthology.orchestra.run.smithi028.stdout: flex-2.6.4-9.el9.x86_64 2024-09-13T16:43:00.441 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T16:43:00.441 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T16:43:00.441 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T16:43:00.441 INFO:teuthology.orchestra.run.smithi028.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-13T16:43:00.442 INFO:teuthology.orchestra.run.smithi028.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-13T16:43:00.442 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T16:43:00.442 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-13T16:43:00.442 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T16:43:00.442 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T16:43:00.442 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.442 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.442 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.442 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T16:43:00.442 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T16:43:00.442 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T16:43:00.442 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T16:43:00.443 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T16:43:00.443 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.443 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.443 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T16:43:00.443 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.443 INFO:teuthology.orchestra.run.smithi028.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-13T16:43:00.443 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T16:43:00.443 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-13T16:43:00.443 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-13T16:43:00.443 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T16:43:00.443 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T16:43:00.443 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T16:43:00.443 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-13T16:43:00.444 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T16:43:00.444 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-13.16-1.el9.x86_64 2024-09-13T16:43:00.444 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-13T16:43:00.444 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-13T16:43:00.444 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-13T16:43:00.444 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T16:43:00.444 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T16:43:00.444 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T16:43:00.444 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T16:43:00.444 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.444 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.444 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.444 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T16:43:00.445 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T16:43:00.445 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T16:43:00.445 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T16:43:00.445 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T16:43:00.445 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T16:43:00.445 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T16:43:00.445 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T16:43:00.445 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T16:43:00.445 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T16:43:00.445 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T16:43:00.445 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T16:43:00.445 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T16:43:00.445 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T16:43:00.446 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T16:43:00.446 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T16:43:00.446 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T16:43:00.446 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T16:43:00.446 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T16:43:00.446 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T16:43:00.446 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T16:43:00.446 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T16:43:00.446 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T16:43:00.446 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T16:43:00.446 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T16:43:00.446 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T16:43:00.447 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T16:43:00.447 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T16:43:00.447 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T16:43:00.447 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T16:43:00.447 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T16:43:00.447 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T16:43:00.447 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T16:43:00.447 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T16:43:00.447 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.447 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.447 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T16:43:00.447 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T16:43:00.447 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T16:43:00.447 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.448 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T16:43:00.448 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T16:43:00.448 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T16:43:00.448 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T16:43:00.448 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T16:43:00.448 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T16:43:00.448 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T16:43:00.448 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T16:43:00.448 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T16:43:00.448 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T16:43:00.448 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T16:43:00.448 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.448 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.449 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T16:43:00.449 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T16:43:00.449 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T16:43:00.449 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T16:43:00.449 INFO:teuthology.orchestra.run.smithi028.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-13T16:43:00.449 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-13T16:43:00.449 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T16:43:00.449 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-13T16:43:00.449 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:00.450 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-13T16:43:01.057 DEBUG:teuthology.parallel:result is None 2024-09-13T16:43:01.058 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T16:43:01.214 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T16:43:01.259 INFO:teuthology.orchestra.run.smithi028.stdout:18.2.4-850.g50bbe7f3.el9 2024-09-13T16:43:01.260 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-850.g50bbe7f3.el9 2024-09-13T16:43:01.260 INFO:teuthology.task.install:The correct ceph version 18.2.4-850.g50bbe7f3 is installed. 2024-09-13T16:43:01.262 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T16:43:01.411 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T16:43:01.437 INFO:teuthology.orchestra.run.smithi191.stdout:18.2.4-850.g50bbe7f3.el9 2024-09-13T16:43:01.438 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-850.g50bbe7f3.el9 2024-09-13T16:43:01.438 INFO:teuthology.task.install:The correct ceph version 18.2.4-850.g50bbe7f3 is installed. 2024-09-13T16:43:01.440 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T16:43:01.440 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:01.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T16:43:01.499 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:01.499 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T16:43:01.531 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T16:43:01.531 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:01.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T16:43:01.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T16:43:01.693 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:01.693 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T16:43:01.723 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T16:43:01.791 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T16:43:01.792 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:01.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T16:43:01.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T16:43:01.951 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:01.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T16:43:01.979 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T16:43:02.047 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T16:43:02.047 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:02.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T16:43:02.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T16:43:02.206 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:02.206 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T16:43:02.234 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T16:43:02.303 INFO:teuthology.run_tasks:Running task ceph... 2024-09-13T16:43:02.385 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-13T16:43:02.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /var/log/ceph 2024-09-13T16:43:02.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /var/log/ceph 2024-09-13T16:43:02.441 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-13T16:43:02.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T16:43:02.445 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T16:43:02.496 INFO:tasks.ceph:Creating extra log directories... 2024-09-13T16:43:02.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T16:43:02.500 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T16:43:02.562 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-09-13T16:43:02.562 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-09-13T16:43:02.562 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-09-13T16:43:02.562 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:02.562 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-13T16:43:02.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-13T16:43:02.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-13T16:43:02.832 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:02.832 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-13T16:43:02.850 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-13T16:43:02.975 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-13T16:43:03.006 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-13T16:43:03.006 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'overall HEALTH_', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-13T16:43:03.007 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903833', 'branch': 'wip-rishabh-testing-20240910.095512-reef', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}}', 'email': None, 'first_in_suite': False, 'job_id': '7903833', 'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-rishabh-testing-20240910.095512-reef'}, 'ceph': {'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'overall HEALTH_', '\\(MON_DOWN\\)'], 'sha1': '50bbe7f395997dcd871813949c49addbbf630361'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '50bbe7f395997dcd871813949c49addbbf630361'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-rishabh-testing-20240910.095512-reef', 'sha1': '50bbe7f395997dcd871813949c49addbbf630361'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 1152, 'sha1': '50bbe7f395997dcd871813949c49addbbf630361', 'sleep_before_teardown': 0, 'subset': '984/1024', 'suite': 'fs', 'suite_branch': 'wip-rishabh-testing-20240910.095512-reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_50bbe7f395997dcd871813949c49addbbf630361/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '50bbe7f395997dcd871813949c49addbbf630361', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi028.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLmqOJH6hrd6+FHScvw0KFTm9tf2H3Cjqo0IzxKbrZ+xzeRQM3ZbE2myVHjp7c6sn5WKtKzVVYwYtJIQWXgOufc=', 'smithi191.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPqJZWX7a+7f4/ul+wjpq2z8cqD/fdqYZyTtAprwnkX1mDrWyiVEz608p6BeRGsGSEnPccSeRTZWBlnnBPaDtmc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '50bbe7f395997dcd871813949c49addbbf630361'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'overall HEALTH_', '\\(MON_DOWN\\)'], 'sha1': '50bbe7f395997dcd871813949c49addbbf630361', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'mon_thrash': {'check_mds_failover': True, 'revive_delay': 20, 'thrash_delay': 10}}, {'workunit': {'clients': {'all': ['suites/iozone.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-13_14:36:41', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-13T16:43:03.007 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T16:43:03.011 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T16:43:03.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T16:43:03.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T16:43:03.145 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:03.145 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T16:43:03.180 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T16:43:03.180 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-09-13T16:43:03.255 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T16:43:03.255 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T16:43:03.255 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 640 Links: 1 2024-09-13T16:43:03.256 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T16:43:03.256 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T16:43:03.256 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-13 16:42:53.315796215 +0000 2024-09-13T16:43:03.256 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-13 16:40:58.959723949 +0000 2024-09-13T16:43:03.256 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-13 16:40:58.959723949 +0000 2024-09-13T16:43:03.256 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-13 16:40:58.959723949 +0000 2024-09-13T16:43:03.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T16:43:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T16:43:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T16:43:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000207626 s, 2.5 MB/s 2024-09-13T16:43:03.350 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T16:43:03.386 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-09-13T16:43:03.461 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T16:43:03.461 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T16:43:03.461 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 658 Links: 1 2024-09-13T16:43:03.461 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T16:43:03.461 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T16:43:03.461 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-13 16:42:53.315796215 +0000 2024-09-13T16:43:03.461 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-13 16:40:59.422711060 +0000 2024-09-13T16:43:03.461 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-13 16:40:59.422711060 +0000 2024-09-13T16:43:03.461 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-13 16:40:59.422711060 +0000 2024-09-13T16:43:03.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T16:43:03.554 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T16:43:03.554 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T16:43:03.554 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000176657 s, 2.9 MB/s 2024-09-13T16:43:03.556 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T16:43:03.593 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-09-13T16:43:03.670 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T16:43:03.670 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T16:43:03.670 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 677 Links: 1 2024-09-13T16:43:03.670 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T16:43:03.670 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T16:43:03.670 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-13 16:42:53.316796187 +0000 2024-09-13T16:43:03.670 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-13 16:40:59.879698338 +0000 2024-09-13T16:43:03.670 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-13 16:40:59.879698338 +0000 2024-09-13T16:43:03.670 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-13 16:40:59.879698338 +0000 2024-09-13T16:43:03.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T16:43:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T16:43:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T16:43:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000169806 s, 3.0 MB/s 2024-09-13T16:43:03.767 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T16:43:03.804 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-09-13T16:43:03.880 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T16:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T16:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 694 Links: 1 2024-09-13T16:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T16:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T16:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-13 16:42:53.317796159 +0000 2024-09-13T16:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-13 16:41:00.339685532 +0000 2024-09-13T16:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-13 16:41:00.339685532 +0000 2024-09-13T16:43:03.881 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-13 16:41:00.339685532 +0000 2024-09-13T16:43:03.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T16:43:03.973 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T16:43:03.973 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T16:43:03.973 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000182585 s, 2.8 MB/s 2024-09-13T16:43:03.976 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T16:43:04.012 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T16:43:04.013 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:04.013 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T16:43:04.032 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T16:43:04.032 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2024-09-13T16:43:04.090 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T16:43:04.090 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T16:43:04.090 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-13T16:43:04.090 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T16:43:04.090 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T16:43:04.090 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-13 16:42:46.437596092 +0000 2024-09-13T16:43:04.091 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-13 16:40:21.348058984 +0000 2024-09-13T16:43:04.091 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-13 16:40:21.348058984 +0000 2024-09-13T16:43:04.091 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-09-13 16:40:21.348058984 +0000 2024-09-13T16:43:04.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T16:43:04.157 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-13T16:43:04.240 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-13T16:43:04.240 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000197654 s, 2.6 MB/s 2024-09-13T16:43:04.242 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T16:43:04.302 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2024-09-13T16:43:04.371 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T16:43:04.372 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T16:43:04.372 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-09-13T16:43:04.372 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T16:43:04.372 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T16:43:04.372 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-13 16:42:46.438596073 +0000 2024-09-13T16:43:04.372 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-13 16:40:21.680052777 +0000 2024-09-13T16:43:04.372 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-13 16:40:21.680052777 +0000 2024-09-13T16:43:04.372 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-09-13 16:40:21.680052777 +0000 2024-09-13T16:43:04.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T16:43:04.448 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-13T16:43:04.448 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-13T16:43:04.448 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000228033 s, 2.2 MB/s 2024-09-13T16:43:04.450 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T16:43:04.516 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2024-09-13T16:43:04.574 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T16:43:04.574 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T16:43:04.574 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-13T16:43:04.574 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T16:43:04.574 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T16:43:04.574 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-13 16:42:46.438596073 +0000 2024-09-13T16:43:04.574 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-13 16:40:22.017046476 +0000 2024-09-13T16:43:04.574 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-13 16:40:22.017046476 +0000 2024-09-13T16:43:04.574 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-09-13 16:40:22.017046476 +0000 2024-09-13T16:43:04.575 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T16:43:04.640 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-13T16:43:04.641 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-13T16:43:04.641 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.00022011 s, 2.3 MB/s 2024-09-13T16:43:04.642 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T16:43:04.700 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2024-09-13T16:43:04.758 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T16:43:04.758 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T16:43:04.758 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-13T16:43:04.758 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T16:43:04.758 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T16:43:04.758 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-13 16:42:46.438596073 +0000 2024-09-13T16:43:04.758 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-13 16:40:22.375039783 +0000 2024-09-13T16:43:04.758 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-13 16:40:22.375039783 +0000 2024-09-13T16:43:04.758 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-09-13 16:40:22.375039783 +0000 2024-09-13T16:43:04.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T16:43:04.825 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-13T16:43:04.825 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-13T16:43:04.825 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.00022536 s, 2.3 MB/s 2024-09-13T16:43:04.825 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T16:43:04.890 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T16:43:04.890 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-13T16:43:04.890 INFO:tasks.ceph:Generating config... 2024-09-13T16:43:04.891 INFO:tasks.ceph:[client] client mount timeout = 600 2024-09-13T16:43:04.891 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-09-13T16:43:04.891 INFO:tasks.ceph:[client] debug client = 20 2024-09-13T16:43:04.892 INFO:tasks.ceph:[client] debug ms = 1 2024-09-13T16:43:04.892 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-09-13T16:43:04.892 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-09-13T16:43:04.892 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-09-13T16:43:04.892 INFO:tasks.ceph:[mds] debug mds = 20 2024-09-13T16:43:04.892 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-09-13T16:43:04.892 INFO:tasks.ceph:[mds] debug ms = 1 2024-09-13T16:43:04.892 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-09-13T16:43:04.892 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-09-13T16:43:04.892 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-09-13T16:43:04.892 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-09-13T16:43:04.892 INFO:tasks.ceph:[mds] mds debug frag = True 2024-09-13T16:43:04.893 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-09-13T16:43:04.893 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2024-09-13T16:43:04.893 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-09-13T16:43:04.893 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-09-13T16:43:04.893 INFO:tasks.ceph:[mds] mds_dir_prefetch = True 2024-09-13T16:43:04.893 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2024-09-13T16:43:04.893 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-09-13T16:43:04.893 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-09-13T16:43:04.893 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-09-13T16:43:04.893 INFO:tasks.ceph:[mgr] debug client = 20 2024-09-13T16:43:04.893 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-13T16:43:04.893 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-13T16:43:04.893 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-13T16:43:04.893 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-13T16:43:04.893 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-13T16:43:04.894 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-09-13T16:43:04.894 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-13T16:43:04.894 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-13T16:43:04.894 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-13T16:43:04.894 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-13T16:43:04.894 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-13T16:43:04.894 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-13T16:43:04.894 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-13T16:43:04.894 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-13T16:43:04.894 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-13T16:43:04.894 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-13T16:43:04.894 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-13T16:43:04.894 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-13T16:43:04.894 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-13T16:43:04.895 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-13T16:43:04.895 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-13T16:43:04.895 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-13T16:43:04.895 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-09-13T16:43:04.895 INFO:tasks.ceph:Setting up mon.a... 2024-09-13T16:43:04.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-13T16:43:04.999 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.keyring 2024-09-13T16:43:05.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-13T16:43:05.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T16:43:05.149 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.28'), ('mon.b', '172.21.15.191'), ('mon.c', '[v2:172.21.15.191:3301,v1:172.21.15.191:6790]')] 2024-09-13T16:43:05.150 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.28,172.21.15.191,[v2:172.21.15.191:3301,v1:172.21.15.191:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-09-13T16:43:05.150 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:05.151 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T16:43:05.187 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.28 --add b 172.21.15.191 --addv c '[v2:172.21.15.191:3301,v1:172.21.15.191:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:43:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:43:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: generated fsid 8e702092-2b7f-4633-9ce6-17040007e53d 2024-09-13T16:43:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = pacific 2024-09-13T16:43:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 0 2024-09-13T16:43:05.306 INFO:teuthology.orchestra.run.smithi028.stdout:fsid 8e702092-2b7f-4633-9ce6-17040007e53d 2024-09-13T16:43:05.307 INFO:teuthology.orchestra.run.smithi028.stdout:last_changed 2024-09-13T16:43:05.305835+0000 2024-09-13T16:43:05.307 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-09-13T16:43:05.305835+0000 2024-09-13T16:43:05.307 INFO:teuthology.orchestra.run.smithi028.stdout:min_mon_release 16 (pacific) 2024-09-13T16:43:05.307 INFO:teuthology.orchestra.run.smithi028.stdout:election_strategy: 1 2024-09-13T16:43:05.307 INFO:teuthology.orchestra.run.smithi028.stdout:0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-13T16:43:05.307 INFO:teuthology.orchestra.run.smithi028.stdout:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2024-09-13T16:43:05.307 INFO:teuthology.orchestra.run.smithi028.stdout:2: [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] mon.c 2024-09-13T16:43:05.307 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-13T16:43:05.309 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T16:43:05.345 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8e702092-2b7f-4633-9ce6-17040007e53d... 2024-09-13T16:43:05.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T16:43:05.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T16:43:05.444 INFO:teuthology.orchestra.run.smithi191.stdout:[global] 2024-09-13T16:43:05.445 INFO:teuthology.orchestra.run.smithi191.stdout: chdir = "" 2024-09-13T16:43:05.445 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T16:43:05.445 INFO:teuthology.orchestra.run.smithi191.stdout: auth supported = cephx 2024-09-13T16:43:05.445 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.445 INFO:teuthology.orchestra.run.smithi191.stdout: filestore xattr use omap = true 2024-09-13T16:43:05.445 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.445 INFO:teuthology.orchestra.run.smithi191.stdout: mon clock drift allowed = 1.000 2024-09-13T16:43:05.445 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.445 INFO:teuthology.orchestra.run.smithi191.stdout: osd crush chooseleaf type = 0 2024-09-13T16:43:05.445 INFO:teuthology.orchestra.run.smithi191.stdout: auth debug = true 2024-09-13T16:43:05.445 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.446 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on old message = true 2024-09-13T16:43:05.446 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on bug = true 2024-09-13T16:43:05.446 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.446 INFO:teuthology.orchestra.run.smithi191.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T16:43:05.446 INFO:teuthology.orchestra.run.smithi191.stdout: mon pg warn max object skew = 0 2024-09-13T16:43:05.446 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.446 INFO:teuthology.orchestra.run.smithi191.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T16:43:05.446 INFO:teuthology.orchestra.run.smithi191.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T16:43:05.446 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.446 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default size = 2 2024-09-13T16:43:05.446 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.447 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow primary affinity = true 2024-09-13T16:43:05.447 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow pg remap = true 2024-09-13T16:43:05.447 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on legacy crush tunables = false 2024-09-13T16:43:05.447 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on crush straw calc version zero = false 2024-09-13T16:43:05.447 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on no sortbitwise = false 2024-09-13T16:43:05.447 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on osd down out interval zero = false 2024-09-13T16:43:05.447 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on too few osds = false 2024-09-13T16:43:05.447 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T16:43:05.447 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T16:43:05.447 INFO:teuthology.orchestra.run.smithi191.stdout: mon_allow_pool_size_one = true 2024-09-13T16:43:05.447 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.447 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T16:43:05.448 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.448 INFO:teuthology.orchestra.run.smithi191.stdout: osd default data pool replay window = 5 2024-09-13T16:43:05.448 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.448 INFO:teuthology.orchestra.run.smithi191.stdout: mon allow pool delete = true 2024-09-13T16:43:05.448 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.448 INFO:teuthology.orchestra.run.smithi191.stdout: mon cluster log file level = debug 2024-09-13T16:43:05.448 INFO:teuthology.orchestra.run.smithi191.stdout: debug asserts on shutdown = true 2024-09-13T16:43:05.448 INFO:teuthology.orchestra.run.smithi191.stdout: mon health detail to clog = false 2024-09-13T16:43:05.448 INFO:teuthology.orchestra.run.smithi191.stdout: mon host = "172.21.15.28,172.21.15.191,[v2:172.21.15.191:3301,v1:172.21.15.191:6790]" 2024-09-13T16:43:05.448 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on skipped message = False 2024-09-13T16:43:05.448 INFO:teuthology.orchestra.run.smithi191.stdout: fsid = 8e702092-2b7f-4633-9ce6-17040007e53d 2024-09-13T16:43:05.448 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.449 INFO:teuthology.orchestra.run.smithi191.stdout:[osd] 2024-09-13T16:43:05.449 INFO:teuthology.orchestra.run.smithi191.stdout: osd journal size = 100 2024-09-13T16:43:05.449 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.449 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub load threshold = 5.0 2024-09-13T16:43:05.449 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub max interval = 600 2024-09-13T16:43:05.449 INFO:teuthology.orchestra.run.smithi191.stdout: osd mclock profile = high_recovery_ops 2024-09-13T16:43:05.449 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.449 INFO:teuthology.orchestra.run.smithi191.stdout: osd recover clone overlap = true 2024-09-13T16:43:05.449 INFO:teuthology.orchestra.run.smithi191.stdout: osd recovery max chunk = 1048576 2024-09-13T16:43:05.449 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.449 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug shutdown = true 2024-09-13T16:43:05.449 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug op order = true 2024-09-13T16:43:05.449 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify stray on activate = true 2024-09-13T16:43:05.450 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.450 INFO:teuthology.orchestra.run.smithi191.stdout: osd open classes on start = true 2024-09-13T16:43:05.450 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug pg log writeout = true 2024-09-13T16:43:05.450 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.450 INFO:teuthology.orchestra.run.smithi191.stdout: osd deep scrub update digest min age = 30 2024-09-13T16:43:05.450 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.450 INFO:teuthology.orchestra.run.smithi191.stdout: osd map max advance = 10 2024-09-13T16:43:05.450 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.450 INFO:teuthology.orchestra.run.smithi191.stdout: journal zero on create = true 2024-09-13T16:43:05.450 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.450 INFO:teuthology.orchestra.run.smithi191.stdout: filestore ondisk finisher threads = 3 2024-09-13T16:43:05.450 INFO:teuthology.orchestra.run.smithi191.stdout: filestore apply finisher threads = 3 2024-09-13T16:43:05.450 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.451 INFO:teuthology.orchestra.run.smithi191.stdout: bdev debug aio = true 2024-09-13T16:43:05.451 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug misdirected ops = true 2024-09-13T16:43:05.451 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore block size = 96636764160 2024-09-13T16:43:05.451 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore compression mode = aggressive 2024-09-13T16:43:05.451 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore fsck on mount = True 2024-09-13T16:43:05.451 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluefs = 20 2024-09-13T16:43:05.451 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluestore = 20 2024-09-13T16:43:05.451 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-13T16:43:05.451 INFO:teuthology.orchestra.run.smithi191.stdout: debug osd = 20 2024-09-13T16:43:05.451 INFO:teuthology.orchestra.run.smithi191.stdout: debug rocksdb = 10 2024-09-13T16:43:05.451 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T16:43:05.451 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd full ratio = 0.9 2024-09-13T16:43:05.452 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd nearfull ratio = 0.8 2024-09-13T16:43:05.452 INFO:teuthology.orchestra.run.smithi191.stdout: osd blocked scrub grace period = 3600 2024-09-13T16:43:05.452 INFO:teuthology.orchestra.run.smithi191.stdout: osd failsafe full ratio = 0.95 2024-09-13T16:43:05.452 INFO:teuthology.orchestra.run.smithi191.stdout: osd max markdown count = 1000 2024-09-13T16:43:05.452 INFO:teuthology.orchestra.run.smithi191.stdout: osd objectstore = bluestore 2024-09-13T16:43:05.452 INFO:teuthology.orchestra.run.smithi191.stdout: osd op complaint time = 180 2024-09-13T16:43:05.452 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.452 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr] 2024-09-13T16:43:05.452 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-13T16:43:05.452 INFO:teuthology.orchestra.run.smithi191.stdout: debug mgr = 20 2024-09-13T16:43:05.452 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-09-13T16:43:05.452 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-09-13T16:43:05.452 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:43:05.453 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:43:05.453 INFO:teuthology.orchestra.run.smithi191.stdout: mgr/telemetry/nag = false 2024-09-13T16:43:05.453 INFO:teuthology.orchestra.run.smithi191.stdout: debug client = 20 2024-09-13T16:43:05.453 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.453 INFO:teuthology.orchestra.run.smithi191.stdout:[mon] 2024-09-13T16:43:05.453 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-13T16:43:05.453 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-09-13T16:43:05.453 INFO:teuthology.orchestra.run.smithi191.stdout: debug paxos = 20 2024-09-13T16:43:05.453 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-09-13T16:43:05.453 INFO:teuthology.orchestra.run.smithi191.stdout: mon data avail warn = 5 2024-09-13T16:43:05.453 INFO:teuthology.orchestra.run.smithi191.stdout: mon mgr mkfs grace = 240 2024-09-13T16:43:05.453 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:43:05.453 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd reporter subtree level = osd 2024-09-13T16:43:05.454 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd prime pg temp = true 2024-09-13T16:43:05.454 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:43:05.454 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.454 INFO:teuthology.orchestra.run.smithi191.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T16:43:05.454 INFO:teuthology.orchestra.run.smithi191.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T16:43:05.454 INFO:teuthology.orchestra.run.smithi191.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T16:43:05.454 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.454 INFO:teuthology.orchestra.run.smithi191.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T16:43:05.454 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T16:43:05.454 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T16:43:05.454 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.454 INFO:teuthology.orchestra.run.smithi191.stdout: # 1m isn't quite enough 2024-09-13T16:43:05.455 INFO:teuthology.orchestra.run.smithi191.stdout: mon_down_mkfs_grace = 2m 2024-09-13T16:43:05.455 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.455 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_filestore_osds = false 2024-09-13T16:43:05.455 INFO:teuthology.orchestra.run.smithi191.stdout: mon op complaint time = 120 2024-09-13T16:43:05.455 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on pool no app = False 2024-09-13T16:43:05.455 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-13T16:43:05.455 INFO:teuthology.orchestra.run.smithi191.stdout:[client] 2024-09-13T16:43:05.455 INFO:teuthology.orchestra.run.smithi191.stdout: rgw cache enabled = true 2024-09-13T16:43:05.455 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable ops log = true 2024-09-13T16:43:05.455 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable usage log = true 2024-09-13T16:43:05.455 INFO:teuthology.orchestra.run.smithi191.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T16:43:05.456 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T16:43:05.456 INFO:teuthology.orchestra.run.smithi191.stdout: client mount timeout = 600 2024-09-13T16:43:05.456 INFO:teuthology.orchestra.run.smithi191.stdout: client_shutdown_timeout = 120 2024-09-13T16:43:05.456 INFO:teuthology.orchestra.run.smithi191.stdout: debug client = 20 2024-09-13T16:43:05.456 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-13T16:43:05.456 INFO:teuthology.orchestra.run.smithi191.stdout: rados mon op timeout = 900 2024-09-13T16:43:05.456 INFO:teuthology.orchestra.run.smithi191.stdout: rados osd op timeout = 900 2024-09-13T16:43:05.456 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.a] 2024-09-13T16:43:05.456 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.b] 2024-09-13T16:43:05.456 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.c] 2024-09-13T16:43:05.456 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.a] 2024-09-13T16:43:05.456 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.c] 2024-09-13T16:43:05.457 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.e] 2024-09-13T16:43:05.457 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.b] 2024-09-13T16:43:05.457 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.d] 2024-09-13T16:43:05.457 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.f] 2024-09-13T16:43:05.457 INFO:teuthology.orchestra.run.smithi191.stdout:[mds] 2024-09-13T16:43:05.457 INFO:teuthology.orchestra.run.smithi191.stdout: debug mds = 20 2024-09-13T16:43:05.457 INFO:teuthology.orchestra.run.smithi191.stdout: debug mds balancer = 20 2024-09-13T16:43:05.457 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-13T16:43:05.457 INFO:teuthology.orchestra.run.smithi191.stdout: mds bal fragment size max = 10000 2024-09-13T16:43:05.457 INFO:teuthology.orchestra.run.smithi191.stdout: mds bal merge size = 5 2024-09-13T16:43:05.457 INFO:teuthology.orchestra.run.smithi191.stdout: mds bal split bits = 3 2024-09-13T16:43:05.458 INFO:teuthology.orchestra.run.smithi191.stdout: mds bal split size = 100 2024-09-13T16:43:05.458 INFO:teuthology.orchestra.run.smithi191.stdout: mds debug frag = True 2024-09-13T16:43:05.458 INFO:teuthology.orchestra.run.smithi191.stdout: mds debug scatterstat = True 2024-09-13T16:43:05.458 INFO:teuthology.orchestra.run.smithi191.stdout: mds oft prefetch dirfrags = True 2024-09-13T16:43:05.458 INFO:teuthology.orchestra.run.smithi191.stdout: mds op complaint time = 180 2024-09-13T16:43:05.458 INFO:teuthology.orchestra.run.smithi191.stdout: mds verify scatter = True 2024-09-13T16:43:05.458 INFO:teuthology.orchestra.run.smithi191.stdout: mds_dir_prefetch = True 2024-09-13T16:43:05.458 INFO:teuthology.orchestra.run.smithi191.stdout: mds_sleep_rank_change = 5000000.0 2024-09-13T16:43:05.458 INFO:teuthology.orchestra.run.smithi191.stdout: osd op complaint time = 180 2024-09-13T16:43:05.458 INFO:teuthology.orchestra.run.smithi191.stdout: rados mon op timeout = 900 2024-09-13T16:43:05.458 INFO:teuthology.orchestra.run.smithi191.stdout: rados osd op timeout = 900 2024-09-13T16:43:05.492 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-09-13T16:43:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: chdir = "" 2024-09-13T16:43:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T16:43:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: auth supported = cephx 2024-09-13T16:43:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: filestore xattr use omap = true 2024-09-13T16:43:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: mon clock drift allowed = 1.000 2024-09-13T16:43:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: osd crush chooseleaf type = 0 2024-09-13T16:43:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: auth debug = true 2024-09-13T16:43:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on old message = true 2024-09-13T16:43:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on bug = true 2024-09-13T16:43:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T16:43:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: mon pg warn max object skew = 0 2024-09-13T16:43:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T16:43:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T16:43:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default size = 2 2024-09-13T16:43:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow primary affinity = true 2024-09-13T16:43:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow pg remap = true 2024-09-13T16:43:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on legacy crush tunables = false 2024-09-13T16:43:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on crush straw calc version zero = false 2024-09-13T16:43:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on no sortbitwise = false 2024-09-13T16:43:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on osd down out interval zero = false 2024-09-13T16:43:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on too few osds = false 2024-09-13T16:43:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T16:43:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T16:43:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: mon_allow_pool_size_one = true 2024-09-13T16:43:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T16:43:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: osd default data pool replay window = 5 2024-09-13T16:43:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: mon allow pool delete = true 2024-09-13T16:43:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: mon cluster log file level = debug 2024-09-13T16:43:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: debug asserts on shutdown = true 2024-09-13T16:43:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: mon health detail to clog = false 2024-09-13T16:43:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: mon host = "172.21.15.28,172.21.15.191,[v2:172.21.15.191:3301,v1:172.21.15.191:6790]" 2024-09-13T16:43:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on skipped message = False 2024-09-13T16:43:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 8e702092-2b7f-4633-9ce6-17040007e53d 2024-09-13T16:43:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.497 INFO:teuthology.orchestra.run.smithi028.stdout:[osd] 2024-09-13T16:43:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: osd journal size = 100 2024-09-13T16:43:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub load threshold = 5.0 2024-09-13T16:43:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub max interval = 600 2024-09-13T16:43:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: osd mclock profile = high_recovery_ops 2024-09-13T16:43:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: osd recover clone overlap = true 2024-09-13T16:43:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: osd recovery max chunk = 1048576 2024-09-13T16:43:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug shutdown = true 2024-09-13T16:43:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug op order = true 2024-09-13T16:43:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug verify stray on activate = true 2024-09-13T16:43:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: osd open classes on start = true 2024-09-13T16:43:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug pg log writeout = true 2024-09-13T16:43:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: osd deep scrub update digest min age = 30 2024-09-13T16:43:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: osd map max advance = 10 2024-09-13T16:43:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: journal zero on create = true 2024-09-13T16:43:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: filestore ondisk finisher threads = 3 2024-09-13T16:43:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: filestore apply finisher threads = 3 2024-09-13T16:43:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: bdev debug aio = true 2024-09-13T16:43:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug misdirected ops = true 2024-09-13T16:43:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore block size = 96636764160 2024-09-13T16:43:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore compression mode = aggressive 2024-09-13T16:43:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore fsck on mount = True 2024-09-13T16:43:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluefs = 20 2024-09-13T16:43:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluestore = 20 2024-09-13T16:43:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T16:43:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: debug osd = 20 2024-09-13T16:43:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: debug rocksdb = 10 2024-09-13T16:43:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T16:43:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd full ratio = 0.9 2024-09-13T16:43:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd nearfull ratio = 0.8 2024-09-13T16:43:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: osd blocked scrub grace period = 3600 2024-09-13T16:43:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: osd failsafe full ratio = 0.95 2024-09-13T16:43:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: osd max markdown count = 1000 2024-09-13T16:43:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: osd objectstore = bluestore 2024-09-13T16:43:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-09-13T16:43:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.502 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr] 2024-09-13T16:43:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T16:43:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: debug mgr = 20 2024-09-13T16:43:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-09-13T16:43:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-09-13T16:43:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:43:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:43:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: mgr/telemetry/nag = false 2024-09-13T16:43:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-09-13T16:43:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.503 INFO:teuthology.orchestra.run.smithi028.stdout:[mon] 2024-09-13T16:43:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T16:43:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-09-13T16:43:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: debug paxos = 20 2024-09-13T16:43:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-09-13T16:43:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: mon data avail warn = 5 2024-09-13T16:43:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: mon mgr mkfs grace = 240 2024-09-13T16:43:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:43:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd reporter subtree level = osd 2024-09-13T16:43:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd prime pg temp = true 2024-09-13T16:43:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:43:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T16:43:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T16:43:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T16:43:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T16:43:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T16:43:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T16:43:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: # 1m isn't quite enough 2024-09-13T16:43:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: mon_down_mkfs_grace = 2m 2024-09-13T16:43:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_filestore_osds = false 2024-09-13T16:43:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: mon op complaint time = 120 2024-09-13T16:43:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on pool no app = False 2024-09-13T16:43:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:05.506 INFO:teuthology.orchestra.run.smithi028.stdout:[client] 2024-09-13T16:43:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: rgw cache enabled = true 2024-09-13T16:43:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable ops log = true 2024-09-13T16:43:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable usage log = true 2024-09-13T16:43:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T16:43:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T16:43:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: client mount timeout = 600 2024-09-13T16:43:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: client_shutdown_timeout = 120 2024-09-13T16:43:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-09-13T16:43:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T16:43:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-09-13T16:43:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-09-13T16:43:05.507 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.a] 2024-09-13T16:43:05.507 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.b] 2024-09-13T16:43:05.507 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.c] 2024-09-13T16:43:05.507 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-09-13T16:43:05.507 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-09-13T16:43:05.507 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.e] 2024-09-13T16:43:05.507 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-09-13T16:43:05.508 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-09-13T16:43:05.508 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.f] 2024-09-13T16:43:05.508 INFO:teuthology.orchestra.run.smithi028.stdout:[mds] 2024-09-13T16:43:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds = 20 2024-09-13T16:43:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds balancer = 20 2024-09-13T16:43:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T16:43:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: mds bal fragment size max = 10000 2024-09-13T16:43:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: mds bal merge size = 5 2024-09-13T16:43:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: mds bal split bits = 3 2024-09-13T16:43:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: mds bal split size = 100 2024-09-13T16:43:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug frag = True 2024-09-13T16:43:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug scatterstat = True 2024-09-13T16:43:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: mds oft prefetch dirfrags = True 2024-09-13T16:43:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: mds op complaint time = 180 2024-09-13T16:43:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: mds verify scatter = True 2024-09-13T16:43:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: mds_dir_prefetch = True 2024-09-13T16:43:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: mds_sleep_rank_change = 5000000.0 2024-09-13T16:43:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-09-13T16:43:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-09-13T16:43:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-09-13T16:43:05.518 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-13T16:43:05.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-13T16:43:05.621 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-13T16:43:05.622 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:05.622 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-13T16:43:05.657 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:05.657 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-13T16:43:05.733 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T16:43:05.734 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:05.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T16:43:05.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T16:43:05.854 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:05.854 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:43:05.929 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi191.front.sepia.ceph.com 2024-09-13T16:43:05.929 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:05.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T16:43:05.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T16:43:05.971 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:05.971 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:43:06.030 INFO:tasks.ceph:Setting up mon nodes... 2024-09-13T16:43:06.030 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-13T16:43:06.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-13T16:43:06.134 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-13T16:43:06.140 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T16:43:06.204 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T16:43:06.207 INFO:tasks.ceph:Setting up mds nodes... 2024-09-13T16:43:06.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-09-13T16:43:06.311 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-09-13T16:43:06.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-09-13T16:43:06.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-09-13T16:43:06.479 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-09-13T16:43:06.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-09-13T16:43:06.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-09-13T16:43:06.644 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-09-13T16:43:06.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-09-13T16:43:06.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-09-13T16:43:06.765 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-09-13T16:43:06.768 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-09-13T16:43:06.795 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-09-13T16:43:06.887 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-09-13T16:43:06.890 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-09-13T16:43:06.915 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-09-13T16:43:07.005 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-09-13T16:43:07.008 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-09-13T16:43:07.032 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-13T16:43:07.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T16:43:07.115 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-13T16:43:07.142 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-13T16:43:07.143 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-13T16:43:07.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-13T16:43:07.199 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T16:43:07.199 INFO:tasks.ceph:role: osd.0 2024-09-13T16:43:07.199 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T16:43:07.199 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T16:43:07.253 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T16:43:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T16:43:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T16:43:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T16:43:07.254 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T16:43:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-09-13T16:43:07.254 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T16:43:07.254 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T16:43:07.254 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T16:43:07.254 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T16:43:07.286 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-09-13T16:43:07.291 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-09-13T16:43:07.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-13T16:43:07.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-13T16:43:07.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-13T16:43:07.490 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T16:43:07.491 INFO:tasks.ceph:role: osd.1 2024-09-13T16:43:07.491 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T16:43:07.491 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T16:43:07.545 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T16:43:07.545 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T16:43:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T16:43:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T16:43:07.546 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T16:43:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-09-13T16:43:07.546 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T16:43:07.546 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T16:43:07.546 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T16:43:07.546 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T16:43:07.578 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-09-13T16:43:07.582 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-09-13T16:43:07.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-13T16:43:07.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-13T16:43:07.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-13T16:43:07.780 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T16:43:07.780 INFO:tasks.ceph:role: osd.2 2024-09-13T16:43:07.780 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T16:43:07.780 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T16:43:07.835 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T16:43:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T16:43:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T16:43:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T16:43:07.836 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T16:43:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-09-13T16:43:07.836 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T16:43:07.836 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T16:43:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T16:43:07.836 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T16:43:07.869 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-09-13T16:43:07.873 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-09-13T16:43:07.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-13T16:43:07.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-13T16:43:08.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-13T16:43:08.069 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T16:43:08.070 INFO:tasks.ceph:role: osd.3 2024-09-13T16:43:08.070 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T16:43:08.070 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T16:43:08.124 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T16:43:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T16:43:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T16:43:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T16:43:08.125 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T16:43:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-09-13T16:43:08.125 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T16:43:08.125 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T16:43:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T16:43:08.125 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T16:43:08.158 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-09-13T16:43:08.161 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-09-13T16:43:08.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-13T16:43:08.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-13T16:43:08.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:43:08.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:08.429+0000 7fb2923ae640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-13T16:43:08.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:08.429+0000 7fb2923ae640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-13T16:43:08.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:08.429+0000 7fb2923ae640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-13T16:43:08.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:08.429+0000 7fb2923ae640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-13T16:43:08.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:08.429+0000 7fb2923ae640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-13T16:43:08.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:08.432+0000 7fb2923ae640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-13T16:43:11.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T16:43:11.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:43:11.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:11.506+0000 7f434fc28640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-13T16:43:11.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:11.506+0000 7f434fc28640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-13T16:43:11.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:11.506+0000 7f434fc28640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-13T16:43:11.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:11.506+0000 7f434fc28640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-13T16:43:11.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:11.506+0000 7f434fc28640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-13T16:43:11.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:11.508+0000 7f434fc28640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-13T16:43:14.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T16:43:14.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:43:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:14.577+0000 7f44d3789640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-13T16:43:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:14.577+0000 7f44d3789640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-13T16:43:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:14.577+0000 7f44d3789640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-13T16:43:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:14.577+0000 7f44d3789640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-13T16:43:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:14.578+0000 7f44d3789640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-13T16:43:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:14.579+0000 7f44d3789640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-13T16:43:17.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T16:43:17.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:43:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:17.667+0000 7f6d6c8d4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-13T16:43:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:17.667+0000 7f6d6c8d4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-13T16:43:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:17.667+0000 7f6d6c8d4640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-13T16:43:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:17.667+0000 7f6d6c8d4640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-13T16:43:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:17.667+0000 7f6d6c8d4640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-13T16:43:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:17.669+0000 7f6d6c8d4640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-13T16:43:20.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T16:43:20.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-13T16:43:20.671 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T16:43:20.671 INFO:tasks.ceph:role: osd.4 2024-09-13T16:43:20.671 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-13T16:43:20.671 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T16:43:20.747 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T16:43:20.747 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T16:43:20.747 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T16:43:20.747 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T16:43:20.747 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T16:43:20.747 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-09-13T16:43:20.747 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T16:43:20.747 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T16:43:20.748 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T16:43:20.748 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T16:43:21.235 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-09-13T16:43:21.237 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-09-13T16:43:21.237 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-13T16:43:21.360 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-13T16:43:21.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-13T16:43:21.459 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T16:43:21.460 INFO:tasks.ceph:role: osd.5 2024-09-13T16:43:21.460 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-13T16:43:21.460 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T16:43:21.532 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T16:43:21.532 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T16:43:21.532 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T16:43:21.532 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T16:43:21.532 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T16:43:21.532 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-09-13T16:43:21.532 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T16:43:21.532 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T16:43:21.533 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T16:43:21.533 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T16:43:21.981 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-09-13T16:43:21.982 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-09-13T16:43:21.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-13T16:43:22.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-13T16:43:22.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-13T16:43:22.188 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T16:43:22.188 INFO:tasks.ceph:role: osd.6 2024-09-13T16:43:22.188 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-13T16:43:22.188 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T16:43:22.260 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T16:43:22.260 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T16:43:22.260 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T16:43:22.260 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T16:43:22.260 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T16:43:22.260 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-09-13T16:43:22.260 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T16:43:22.260 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T16:43:22.261 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T16:43:22.261 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T16:43:22.743 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-09-13T16:43:22.745 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-09-13T16:43:22.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-13T16:43:22.839 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-13T16:43:22.869 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-13T16:43:22.933 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T16:43:22.934 INFO:tasks.ceph:role: osd.7 2024-09-13T16:43:22.934 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-13T16:43:22.934 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T16:43:23.007 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T16:43:23.007 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T16:43:23.007 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T16:43:23.007 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T16:43:23.007 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T16:43:23.007 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-09-13T16:43:23.007 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T16:43:23.007 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T16:43:23.008 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T16:43:23.008 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T16:43:23.520 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-09-13T16:43:23.522 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-09-13T16:43:23.522 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-13T16:43:23.654 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-13T16:43:23.687 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:43:23.784 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:23.782+0000 7f398320b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-13T16:43:23.790 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:23.788+0000 7f398320b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-13T16:43:23.790 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:23.788+0000 7f398320b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-13T16:43:23.790 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:23.788+0000 7f398320b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-13T16:43:23.790 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:23.788+0000 7f398320b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-13T16:43:23.791 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:23.789+0000 7f398320b640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-13T16:43:26.651 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-13T16:43:26.679 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:43:26.769 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:26.766+0000 7ff014f45640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-13T16:43:26.769 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:26.767+0000 7ff014f45640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-13T16:43:26.769 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:26.767+0000 7ff014f45640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-13T16:43:26.770 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:26.767+0000 7ff014f45640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-13T16:43:26.770 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:26.767+0000 7ff014f45640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-13T16:43:26.770 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:26.767+0000 7ff014f45640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-13T16:43:29.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-13T16:43:29.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:43:29.731 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:29.729+0000 7f15621fd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-13T16:43:29.732 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:29.729+0000 7f15621fd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-13T16:43:29.732 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:29.729+0000 7f15621fd640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-13T16:43:29.732 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:29.729+0000 7f15621fd640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-13T16:43:29.732 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:29.729+0000 7f15621fd640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-13T16:43:29.732 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:29.730+0000 7f15621fd640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-13T16:43:32.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-13T16:43:32.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:43:32.709 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:32.707+0000 7f20ebba8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-13T16:43:32.709 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:32.707+0000 7f20ebba8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-13T16:43:32.710 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:32.707+0000 7f20ebba8640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-13T16:43:32.710 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:32.707+0000 7f20ebba8640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-13T16:43:32.710 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:32.707+0000 7f20ebba8640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-13T16:43:32.710 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:32.709+0000 7f20ebba8640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-13T16:43:33.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:43:33.014 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:43:35.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-13T16:43:35.600 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-13T16:43:35.600 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:35.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-13T16:43:35.672 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:35.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-09-13T16:43:35.774 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:35.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-09-13T16:43:35.876 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:35.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-09-13T16:43:35.975 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:35.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-13T16:43:36.071 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:36.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-13T16:43:36.167 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:36.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-13T16:43:36.265 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:36.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-13T16:43:36.363 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:36.363 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-13T16:43:36.395 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:36.395 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-09-13T16:43:36.465 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:36.465 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-09-13T16:43:36.533 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:36.533 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-09-13T16:43:36.601 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:36.601 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-13T16:43:36.670 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:36.670 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-13T16:43:36.737 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:36.738 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-13T16:43:36.805 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:36.805 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-13T16:43:36.873 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:36.873 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-13T16:43:36.909 INFO:tasks.ceph:Adding keys to all mons... 2024-09-13T16:43:36.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T16:43:36.953 DEBUG:teuthology.orchestra.run.smithi191:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T16:43:36.978 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.y] 2024-09-13T16:43:36.978 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCaa+Rmhk0DCBAAycVTTmMNA/JN+iWdVsSSVA== 2024-09-13T16:43:36.979 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.a] 2024-09-13T16:43:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCaa+RmSNuREhAAsoDjyegQjMbKmtz25gCPww== 2024-09-13T16:43:36.979 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.c] 2024-09-13T16:43:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCaa+RmpMCFHBAAwi3k8AOIVImGgPu2a4ecFw== 2024-09-13T16:43:36.979 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.e] 2024-09-13T16:43:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCaa+RmCX1fJhAACkTwLw3V8pnp9kwe4I9qFw== 2024-09-13T16:43:36.979 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.0] 2024-09-13T16:43:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCca+RmDdahGRAAtr+zw1UWK2475NGQDMmSog== 2024-09-13T16:43:36.979 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.1] 2024-09-13T16:43:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCfa+RmRvY9HhAAE7Fu/Dib1vFCUfHYyViEfQ== 2024-09-13T16:43:36.979 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.2] 2024-09-13T16:43:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCia+RmUB97IhAA1bdpwlTqfmMxNmuT2sF/gw== 2024-09-13T16:43:36.979 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.3] 2024-09-13T16:43:36.980 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCla+RmzobTJxAAd+bw0O4ugtrWVEe5wyussQ== 2024-09-13T16:43:36.980 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.x] 2024-09-13T16:43:36.980 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCaa+RmHrkuDBAAdrx6r+T650xo75GW0+/1hw== 2024-09-13T16:43:36.980 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.b] 2024-09-13T16:43:36.980 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCaa+RmgIeULRAAb5QVmgNMKuQW7BFA61xVQg== 2024-09-13T16:43:36.980 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.d] 2024-09-13T16:43:36.980 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCaa+RmEy7lNBAAx8aiMrItvG47mQNmGLiahg== 2024-09-13T16:43:36.980 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.f] 2024-09-13T16:43:36.980 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCba+RmOFtRABAAValZLukjci6RkX7nfE6W8g== 2024-09-13T16:43:36.980 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.4] 2024-09-13T16:43:36.980 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCra+RmSB/CLhAA5VqivZglXCha0nV6lpl89w== 2024-09-13T16:43:36.980 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.5] 2024-09-13T16:43:36.980 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCua+Rm8wfRLRAAq6EjMpBU+HvAfzy2KmrJmw== 2024-09-13T16:43:36.981 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.6] 2024-09-13T16:43:36.981 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCxa+RmL9+TKxAArBT+GgnSzhvMQ7MQNzcA8Q== 2024-09-13T16:43:36.981 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.7] 2024-09-13T16:43:36.981 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQC0a+RmTi9CKhAAkusKJx3WrFqSxD/ByJ8/sQ== 2024-09-13T16:43:36.981 INFO:teuthology.orchestra.run.smithi191.stdout:[client.0] 2024-09-13T16:43:36.981 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCba+RmsDbWBhAAlDZRQVwOQA9YO3aNgbuhmA== 2024-09-13T16:43:37.003 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr.y] 2024-09-13T16:43:37.003 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCaa+Rmhk0DCBAAycVTTmMNA/JN+iWdVsSSVA== 2024-09-13T16:43:37.004 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-09-13T16:43:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCaa+RmSNuREhAAsoDjyegQjMbKmtz25gCPww== 2024-09-13T16:43:37.004 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-09-13T16:43:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCaa+RmpMCFHBAAwi3k8AOIVImGgPu2a4ecFw== 2024-09-13T16:43:37.004 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.e] 2024-09-13T16:43:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCaa+RmCX1fJhAACkTwLw3V8pnp9kwe4I9qFw== 2024-09-13T16:43:37.004 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.0] 2024-09-13T16:43:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCca+RmDdahGRAAtr+zw1UWK2475NGQDMmSog== 2024-09-13T16:43:37.004 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.1] 2024-09-13T16:43:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCfa+RmRvY9HhAAE7Fu/Dib1vFCUfHYyViEfQ== 2024-09-13T16:43:37.004 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.2] 2024-09-13T16:43:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCia+RmUB97IhAA1bdpwlTqfmMxNmuT2sF/gw== 2024-09-13T16:43:37.004 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.3] 2024-09-13T16:43:37.004 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCla+RmzobTJxAAd+bw0O4ugtrWVEe5wyussQ== 2024-09-13T16:43:37.005 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr.x] 2024-09-13T16:43:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCaa+RmHrkuDBAAdrx6r+T650xo75GW0+/1hw== 2024-09-13T16:43:37.005 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-09-13T16:43:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCaa+RmgIeULRAAb5QVmgNMKuQW7BFA61xVQg== 2024-09-13T16:43:37.005 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-09-13T16:43:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCaa+RmEy7lNBAAx8aiMrItvG47mQNmGLiahg== 2024-09-13T16:43:37.005 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.f] 2024-09-13T16:43:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCba+RmOFtRABAAValZLukjci6RkX7nfE6W8g== 2024-09-13T16:43:37.005 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.4] 2024-09-13T16:43:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCra+RmSB/CLhAA5VqivZglXCha0nV6lpl89w== 2024-09-13T16:43:37.005 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.5] 2024-09-13T16:43:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCua+Rm8wfRLRAAq6EjMpBU+HvAfzy2KmrJmw== 2024-09-13T16:43:37.005 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.6] 2024-09-13T16:43:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCxa+RmL9+TKxAArBT+GgnSzhvMQ7MQNzcA8Q== 2024-09-13T16:43:37.006 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.7] 2024-09-13T16:43:37.006 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQC0a+RmTi9CKhAAkusKJx3WrFqSxD/ByJ8/sQ== 2024-09-13T16:43:37.006 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-13T16:43:37.006 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCba+RmsDbWBhAAlDZRQVwOQA9YO3aNgbuhmA== 2024-09-13T16:43:37.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T16:43:37.051 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T16:43:37.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:43:37.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:43:37.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:43:37.242 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:43:37.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:43:37.339 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:43:37.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:43:37.436 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:43:37.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:43:37.529 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:43:37.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:43:37.623 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:43:37.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:43:37.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:43:37.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T16:43:37.808 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T16:43:37.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:43:37.902 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:43:37.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:43:37.999 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:43:38.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:43:38.096 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:43:38.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:43:38.191 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:43:38.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:43:38.285 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:43:38.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:43:38.379 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:43:38.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:43:38.474 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:43:38.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T16:43:38.568 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T16:43:38.661 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-13T16:43:38.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-13T16:43:38.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T16:43:38.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-13T16:43:38.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-13T16:43:38.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T16:43:39.004 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-13T16:43:39.030 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-13T16:43:39.095 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T16:43:39.193 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-13T16:43:39.218 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:43:39.222 DEBUG:teuthology.orchestra.run.smithi191:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:43:39.273 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-13T16:43:39.273 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-13T16:43:39.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-13T16:43:39.298 INFO:tasks.ceph.mon.a:Started 2024-09-13T16:43:39.299 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-13T16:43:39.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-13T16:43:39.315 INFO:tasks.ceph.mon.b:Started 2024-09-13T16:43:39.316 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-13T16:43:39.316 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-13T16:43:39.318 INFO:tasks.ceph.mon.c:Started 2024-09-13T16:43:39.318 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-13T16:43:39.318 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-13T16:43:39.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-13T16:43:39.322 INFO:tasks.ceph.mgr.y:Started 2024-09-13T16:43:39.322 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-13T16:43:39.322 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-13T16:43:39.325 INFO:tasks.ceph.mgr.x:Started 2024-09-13T16:43:39.325 INFO:tasks.ceph:Setting crush tunables to default 2024-09-13T16:43:39.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph osd crush tunables default 2024-09-13T16:43:39.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.477+0000 7f3843200640 1 Processor -- start 2024-09-13T16:43:39.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.477+0000 7f3843200640 1 -- start start 2024-09-13T16:43:39.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.478+0000 7f3843200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f383c06b140 0x7f383c0c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:39.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.478+0000 7f3843200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f383c159730 0x7f383c152a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:39.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.478+0000 7f3843200640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c15ccf0 con 0x7f383c06ad60 2024-09-13T16:43:39.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.479+0000 7f3843200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f383c0cc300 con 0x7f383c159730 2024-09-13T16:43:39.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.479+0000 7f3843200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0cb330 con 0x7f383c06b140 2024-09-13T16:43:39.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.479+0000 7f3833e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f383c06b140 0x7f383c0c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:39.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.479+0000 7f383be00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f383c06ad60 0x7f383c0c9d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:53796/0 (socket says 172.21.15.28:53796) 2024-09-13T16:43:39.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.479+0000 7f383be00640 1 -- 172.21.15.28:0/1450199816 learned_addr learned my addr 172.21.15.28:0/1450199816 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:43:39.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.479+0000 7f3833e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f383c06b140 0x7f383c0c92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:35918/0 (socket says 172.21.15.28:35918) 2024-09-13T16:43:39.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.479+0000 7f3841200640 1 --2- 172.21.15.28:0/1450199816 >> v2:172.21.15.191:3300/0 conn(0x7f383c159730 0x7f383c152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:39.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.479+0000 7f383b400640 1 -- 172.21.15.28:0/1450199816 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993586555 0 0) 0x7f383c15ccf0 con 0x7f383c06ad60 2024-09-13T16:43:39.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.479+0000 7f383b400640 1 -- 172.21.15.28:0/1450199816 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3820003520 con 0x7f383c06ad60 2024-09-13T16:43:39.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.479+0000 7f3841200640 1 -- 172.21.15.28:0/1450199816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f383c06b140 msgr2=0x7f383c0c92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:39.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.479+0000 7f3841200640 1 --2- 172.21.15.28:0/1450199816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f383c06b140 0x7f383c0c92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:39.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.479+0000 7f3841200640 1 -- 172.21.15.28:0/1450199816 >> v1:172.21.15.191:6789/0 conn(0x7f383c06ad60 legacy=0x7f383c0c9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:39.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.479+0000 7f3841200640 1 -- 172.21.15.28:0/1450199816 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c159be0 con 0x7f383c159730 2024-09-13T16:43:39.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.479+0000 7f3833e00640 1 --2- 172.21.15.28:0/1450199816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f383c06b140 0x7f383c0c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.480+0000 7f3841200640 1 --2- 172.21.15.28:0/1450199816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f383c159730 0x7f383c152a40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3834009df0 tx=0x7f383402a750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2113f141c438e6a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.480+0000 7f383b400640 1 -- 172.21.15.28:0/1450199816 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834007740 con 0x7f383c159730 2024-09-13T16:43:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.481+0000 7f383b400640 1 -- 172.21.15.28:0/1450199816 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383402d050 con 0x7f383c159730 2024-09-13T16:43:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.481+0000 7f383b400640 1 -- 172.21.15.28:0/1450199816 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834007ac0 con 0x7f383c159730 2024-09-13T16:43:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.481+0000 7f3843200640 1 -- 172.21.15.28:0/1450199816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f383c159730 msgr2=0x7f383c152a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.481+0000 7f3843200640 1 --2- 172.21.15.28:0/1450199816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f383c159730 0x7f383c152a40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3834009df0 tx=0x7f383402a750 comp rx=0 tx=0).stop 2024-09-13T16:43:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.481+0000 7f3843200640 1 -- 172.21.15.28:0/1450199816 shutdown_connections 2024-09-13T16:43:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.481+0000 7f3843200640 1 --2- 172.21.15.28:0/1450199816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f383c159730 0x7f383c152a40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.481+0000 7f3843200640 1 --2- 172.21.15.28:0/1450199816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f383c06b140 0x7f383c0c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.481+0000 7f3843200640 1 -- 172.21.15.28:0/1450199816 >> 172.21.15.28:0/1450199816 conn(0x7f383c06bfc0 msgr2=0x7f383c0715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.481+0000 7f3843200640 1 -- 172.21.15.28:0/1450199816 shutdown_connections 2024-09-13T16:43:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.481+0000 7f3843200640 1 -- 172.21.15.28:0/1450199816 wait complete. 2024-09-13T16:43:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.483+0000 7f3843200640 1 Processor -- start 2024-09-13T16:43:39.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.483+0000 7f3843200640 1 -- start start 2024-09-13T16:43:39.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.484+0000 7f3843200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f383c06ad60 0x7f383c0cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:39.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.484+0000 7f3843200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f383c06b140 0x7f383c1f5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:39.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.484+0000 7f3843200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383c159730 0x7f383c1f6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:39.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.484+0000 7f3843200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0ceb90 con 0x7f383c159730 2024-09-13T16:43:39.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.484+0000 7f3843200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0ccd20 con 0x7f383c06ad60 2024-09-13T16:43:39.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.484+0000 7f3843200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0cede0 con 0x7f383c06b140 2024-09-13T16:43:39.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.484+0000 7f3833e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f383c06b140 0x7f383c1f5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:39.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.484+0000 7f383be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f383c06ad60 0x7f383c0cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:39.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.484+0000 7f3841200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383c159730 0x7f383c1f6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:39.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.484+0000 7f3833e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f383c06b140 0x7f383c1f5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:35930/0 (socket says 172.21.15.28:35930) 2024-09-13T16:43:39.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.484+0000 7f3833e00640 1 -- 172.21.15.28:0/940235543 learned_addr learned my addr 172.21.15.28:0/940235543 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:39.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.484+0000 7f3833e00640 1 -- 172.21.15.28:0/940235543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f383c06ad60 msgr2=0x7f383c0cd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:39.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.484+0000 7f3833e00640 1 --2- 172.21.15.28:0/940235543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f383c06ad60 0x7f383c0cd1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:39.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.484+0000 7f3833e00640 1 -- 172.21.15.28:0/940235543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383c159730 msgr2=0x7f383c1f6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:39.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.484+0000 7f3833e00640 1 --2- 172.21.15.28:0/940235543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383c159730 0x7f383c1f6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:39.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.484+0000 7f3833e00640 1 -- 172.21.15.28:0/940235543 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3824009400 con 0x7f383c06b140 2024-09-13T16:43:39.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.485+0000 7f3833e00640 1 --2- 172.21.15.28:0/940235543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f383c06b140 0x7f383c1f5c00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f382400e980 tx=0x7f382400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:39.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.485+0000 7f3838c00640 1 -- 172.21.15.28:0/940235543 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382400ca00 con 0x7f383c06b140 2024-09-13T16:43:39.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.485+0000 7f3838c00640 1 -- 172.21.15.28:0/940235543 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382400cb60 con 0x7f383c06b140 2024-09-13T16:43:39.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.485+0000 7f3838c00640 1 -- 172.21.15.28:0/940235543 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3824013470 con 0x7f383c06b140 2024-09-13T16:43:39.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.485+0000 7f3843200640 1 -- 172.21.15.28:0/940235543 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3834009a70 con 0x7f383c06b140 2024-09-13T16:43:39.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.485+0000 7f3843200640 1 -- 172.21.15.28:0/940235543 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383c0cc6d0 con 0x7f383c06b140 2024-09-13T16:43:39.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.485+0000 7f3838c00640 1 -- 172.21.15.28:0/940235543 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f3824005c90 con 0x7f383c06b140 2024-09-13T16:43:39.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.485+0000 7f3838c00640 1 -- 172.21.15.28:0/940235543 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3824013910 con 0x7f383c06b140 2024-09-13T16:43:39.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.486+0000 7f3843200640 1 -- 172.21.15.28:0/940235543 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383c0cf300 con 0x7f383c06b140 2024-09-13T16:43:39.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.488+0000 7f3838c00640 1 -- 172.21.15.28:0/940235543 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f383c0cf300 con 0x7f383c06b140 2024-09-13T16:43:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.554+0000 7f3843200640 1 -- 172.21.15.28:0/940235543 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f383c0cce90 con 0x7f383c06b140 2024-09-13T16:43:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.562+0000 7f3838c00640 1 -- 172.21.15.28:0/940235543 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f3824029020 con 0x7f383c06b140 2024-09-13T16:43:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-09-13T16:43:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.563+0000 7f3843200640 1 -- 172.21.15.28:0/940235543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f383c06b140 msgr2=0x7f383c1f5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.563+0000 7f3843200640 1 --2- 172.21.15.28:0/940235543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f383c06b140 0x7f383c1f5c00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f382400e980 tx=0x7f382400ee40 comp rx=0 tx=0).stop 2024-09-13T16:43:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.563+0000 7f3843200640 1 -- 172.21.15.28:0/940235543 shutdown_connections 2024-09-13T16:43:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.563+0000 7f3843200640 1 --2- 172.21.15.28:0/940235543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383c159730 0x7f383c1f6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.563+0000 7f3843200640 1 --2- 172.21.15.28:0/940235543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f383c06b140 0x7f383c1f5c00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.563+0000 7f3843200640 1 --2- 172.21.15.28:0/940235543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f383c06ad60 0x7f383c0cd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.563+0000 7f3843200640 1 -- 172.21.15.28:0/940235543 >> 172.21.15.28:0/940235543 conn(0x7f383c06bfc0 msgr2=0x7f383c14f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.563+0000 7f3843200640 1 -- 172.21.15.28:0/940235543 shutdown_connections 2024-09-13T16:43:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:39.563+0000 7f3843200640 1 -- 172.21.15.28:0/940235543 wait complete. 2024-09-13T16:43:39.579 INFO:tasks.ceph:check_enable_crimson: False 2024-09-13T16:43:39.579 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-13T16:43:39.579 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:39.579 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-13T16:43:39.635 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:39.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-13T16:43:39.692 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:39.691+0000 7f98e4c38040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T16:43:39.732 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:39.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-13T16:43:39.737 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:39.731+0000 7f98e4c38040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T16:43:39.783 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:39.782+0000 7f26d0638040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T16:43:39.833 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:43:39.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-13T16:43:39.837 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:39.835+0000 7f26d0638040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T16:43:39.874 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:39.873+0000 7f98e4c38040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T16:43:39.895 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:39.895 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-13T16:43:39.921 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:39.921 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-13T16:43:39.926 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:39.925+0000 7f98e4c38040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T16:43:39.950 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:39.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-13T16:43:40.015 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-13T16:43:40.016 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-13T16:43:40.024 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:40.023+0000 7f26d0638040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T16:43:40.081 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 274e98e1-663e-4814-90ad-c7b3026b276d 0 2024-09-13T16:43:40.085 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:40.083+0000 7f26d0638040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T16:43:40.093 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:40.092+0000 7f98e4c38040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T16:43:40.191 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.190+0000 7fce09ff4640 1 Processor -- start 2024-09-13T16:43:40.191 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.190+0000 7fce09ff4640 1 -- start start 2024-09-13T16:43:40.192 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.190+0000 7fce09ff4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fce040c71d0 0x7fce04158b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.192 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.190+0000 7fce09ff4640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce040c5150 con 0x7fce04152d00 2024-09-13T16:43:40.192 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.190+0000 7fce09ff4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce04070ed0 con 0x7fce040c54f0 2024-09-13T16:43:40.192 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.190+0000 7fce09ff4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce0406a210 con 0x7fce040c71d0 2024-09-13T16:43:40.192 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.191+0000 7fcdfbfff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fce040c71d0 0x7fce04158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.193 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.191+0000 7fcdfbfff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fce040c71d0 0x7fce04158b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59742/0 (socket says 172.21.15.191:59742) 2024-09-13T16:43:40.193 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.191+0000 7fcdfbfff640 1 -- 172.21.15.191:0/3437434035 learned_addr learned my addr 172.21.15.191:0/3437434035 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-09-13T16:43:40.193 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.191+0000 7fcdfbfff640 1 -- 172.21.15.191:0/3437434035 >> v1:172.21.15.191:6789/0 conn(0x7fce040c54f0 legacy=0x7fce04163470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.193 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.191+0000 7fcdfbfff640 1 -- 172.21.15.191:0/3437434035 >> v1:172.21.15.28:6789/0 conn(0x7fce04152d00 legacy=0x7fce040cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.193 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.191+0000 7fcdfbfff640 1 -- 172.21.15.191:0/3437434035 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce04071190 con 0x7fce040c71d0 2024-09-13T16:43:40.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.192+0000 7fcdfbfff640 1 --2- 172.21.15.191:0/3437434035 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce040c71d0 0x7fce04158b10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009870 tx=0x7fcdec02a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=445dfe805c685db9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:40.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.192+0000 7fce02ffd640 1 -- 172.21.15.191:0/3437434035 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec032640 con 0x7fce040c71d0 2024-09-13T16:43:40.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.192+0000 7fce02ffd640 1 -- 172.21.15.191:0/3437434035 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdec0327a0 con 0x7fce040c71d0 2024-09-13T16:43:40.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.192+0000 7fce02ffd640 1 -- 172.21.15.191:0/3437434035 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec032b30 con 0x7fce040c71d0 2024-09-13T16:43:40.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.192+0000 7fce09ff4640 1 -- 172.21.15.191:0/3437434035 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce040c71d0 msgr2=0x7fce04158b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.192+0000 7fce09ff4640 1 --2- 172.21.15.191:0/3437434035 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce040c71d0 0x7fce04158b10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009870 tx=0x7fcdec02a300 comp rx=0 tx=0).stop 2024-09-13T16:43:40.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.192+0000 7fce09ff4640 1 -- 172.21.15.191:0/3437434035 shutdown_connections 2024-09-13T16:43:40.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.192+0000 7fce09ff4640 1 --2- 172.21.15.191:0/3437434035 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce040c71d0 0x7fce04158b10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.192+0000 7fce09ff4640 1 -- 172.21.15.191:0/3437434035 >> 172.21.15.191:0/3437434035 conn(0x7fce0406d470 msgr2=0x7fce0406af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:40.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.193+0000 7fce09ff4640 1 -- 172.21.15.191:0/3437434035 shutdown_connections 2024-09-13T16:43:40.195 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.193+0000 7fce09ff4640 1 -- 172.21.15.191:0/3437434035 wait complete. 2024-09-13T16:43:40.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.194+0000 7fce09ff4640 1 Processor -- start 2024-09-13T16:43:40.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.194+0000 7fce09ff4640 1 -- start start 2024-09-13T16:43:40.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.194+0000 7fce09ff4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce040c54f0 0x7fce0414a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.194+0000 7fce09ff4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce040c71d0 0x7fce04153ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.194+0000 7fce09ff4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fce04152d00 0x7fce04154010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.194+0000 7fce09ff4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0414ae50 con 0x7fce040c54f0 2024-09-13T16:43:40.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.194+0000 7fce09ff4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce04145820 con 0x7fce040c71d0 2024-09-13T16:43:40.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.194+0000 7fce09ff4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce04158260 con 0x7fce04152d00 2024-09-13T16:43:40.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.194+0000 7fcdfbfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce040c71d0 0x7fce04153ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.194+0000 7fce03fff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fce04152d00 0x7fce04154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.194+0000 7fcdfbfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce040c71d0 0x7fce04153ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59744/0 (socket says 172.21.15.191:59744) 2024-09-13T16:43:40.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.194+0000 7fcdfbfff640 1 -- 172.21.15.191:0/967791369 learned_addr learned my addr 172.21.15.191:0/967791369 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-09-13T16:43:40.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.195+0000 7fce037fe640 1 --2- 172.21.15.191:0/967791369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce040c54f0 0x7fce0414a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.195+0000 7fcdfbfff640 1 -- 172.21.15.191:0/967791369 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fce04152d00 msgr2=0x7fce04154010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.195+0000 7fcdfbfff640 1 --2- 172.21.15.191:0/967791369 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fce04152d00 0x7fce04154010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.195+0000 7fcdfbfff640 1 -- 172.21.15.191:0/967791369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce040c54f0 msgr2=0x7fce0414a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.195+0000 7fcdfbfff640 1 --2- 172.21.15.191:0/967791369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce040c54f0 0x7fce0414a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.195+0000 7fcdfbfff640 1 -- 172.21.15.191:0/967791369 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec0094d0 con 0x7fce040c71d0 2024-09-13T16:43:40.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.195+0000 7fce03fff640 1 --2- 172.21.15.191:0/967791369 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fce04152d00 0x7fce04154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:40.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.195+0000 7fce037fe640 1 --2- 172.21.15.191:0/967791369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce040c54f0 0x7fce0414a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:43:40.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.195+0000 7fcdfbfff640 1 --2- 172.21.15.191:0/967791369 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce040c71d0 0x7fce04153ad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcdec000c00 tx=0x7fcdec0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:40.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.195+0000 7fce00ff9640 1 -- 172.21.15.191:0/967791369 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec030b80 con 0x7fce040c71d0 2024-09-13T16:43:40.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.195+0000 7fce00ff9640 1 -- 172.21.15.191:0/967791369 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdec030d00 con 0x7fce040c71d0 2024-09-13T16:43:40.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.195+0000 7fce09ff4640 1 -- 172.21.15.191:0/967791369 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce04158d50 con 0x7fce040c71d0 2024-09-13T16:43:40.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.195+0000 7fce00ff9640 1 -- 172.21.15.191:0/967791369 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec03d660 con 0x7fce040c71d0 2024-09-13T16:43:40.199 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.195+0000 7fce09ff4640 1 -- 172.21.15.191:0/967791369 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce04153640 con 0x7fce040c71d0 2024-09-13T16:43:40.200 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.195+0000 7fce00ff9640 1 -- 172.21.15.191:0/967791369 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fcdec039040 con 0x7fce040c71d0 2024-09-13T16:43:40.200 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.195+0000 7fce00ff9640 1 -- 172.21.15.191:0/967791369 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcdec046a10 con 0x7fce040c71d0 2024-09-13T16:43:40.200 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.196+0000 7fce09ff4640 1 -- 172.21.15.191:0/967791369 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce041537d0 con 0x7fce040c71d0 2024-09-13T16:43:40.200 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.199+0000 7fce00ff9640 1 -- 172.21.15.191:0/967791369 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fcdec04b060 con 0x7fce040c71d0 2024-09-13T16:43:40.210 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:40.208+0000 7f98e4c38040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T16:43:40.268 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.267+0000 7fce09ff4640 1 -- 172.21.15.191:0/967791369 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "274e98e1-663e-4814-90ad-c7b3026b276d", "id": 0} v 0) v1 -- 0x7fce041530e0 con 0x7fce040c71d0 2024-09-13T16:43:40.272 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.271+0000 7fce00ff9640 1 -- 172.21.15.191:0/967791369 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "274e98e1-663e-4814-90ad-c7b3026b276d", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcdec041070 con 0x7fce040c71d0 2024-09-13T16:43:40.272 INFO:teuthology.orchestra.run.smithi191.stdout:0 2024-09-13T16:43:40.273 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.272+0000 7fce09ff4640 1 -- 172.21.15.191:0/967791369 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce040c71d0 msgr2=0x7fce04153ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.273 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.272+0000 7fce09ff4640 1 --2- 172.21.15.191:0/967791369 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce040c71d0 0x7fce04153ad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcdec000c00 tx=0x7fcdec0046c0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.273 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.272+0000 7fce09ff4640 1 -- 172.21.15.191:0/967791369 shutdown_connections 2024-09-13T16:43:40.274 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.272+0000 7fce09ff4640 1 --2- 172.21.15.191:0/967791369 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fce04152d00 0x7fce04154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.274 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.272+0000 7fce09ff4640 1 --2- 172.21.15.191:0/967791369 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce040c71d0 0x7fce04153ad0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.274 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.272+0000 7fce09ff4640 1 --2- 172.21.15.191:0/967791369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce040c54f0 0x7fce0414a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.274 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.272+0000 7fce09ff4640 1 -- 172.21.15.191:0/967791369 >> 172.21.15.191:0/967791369 conn(0x7fce0406d470 msgr2=0x7fce040cacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:40.274 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.272+0000 7fce09ff4640 1 -- 172.21.15.191:0/967791369 shutdown_connections 2024-09-13T16:43:40.274 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.272+0000 7fce09ff4640 1 -- 172.21.15.191:0/967791369 wait complete. 2024-09-13T16:43:40.284 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 67a0a7cd-ee9b-471c-a34f-d4fe97186aa0 1 2024-09-13T16:43:40.290 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:40.289+0000 7f26d0638040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T16:43:40.382 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.380+0000 7feac9fbc640 1 Processor -- start 2024-09-13T16:43:40.382 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.380+0000 7feac9fbc640 1 -- start start 2024-09-13T16:43:40.389 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.381+0000 7feac9fbc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feac40c6b10 0x7feac4155b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.389 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.381+0000 7feac9fbc640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac4155680 con 0x7feac414f860 2024-09-13T16:43:40.390 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.381+0000 7feac9fbc640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac40c60c0 con 0x7feac40c2900 2024-09-13T16:43:40.390 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.381+0000 7feac9fbc640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feac40cb4d0 con 0x7feac40c6b10 2024-09-13T16:43:40.390 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.381+0000 7feac2ffd640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7feac40c2900 0x7feac40c6400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:37170/0 (socket says 172.21.15.191:37170) 2024-09-13T16:43:40.390 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.381+0000 7feac2ffd640 1 -- 172.21.15.191:0/4210139000 learned_addr learned my addr 172.21.15.191:0/4210139000 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-09-13T16:43:40.390 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac27fc640 1 -- 172.21.15.191:0/4210139000 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128708481 0 0) 0x7feac40c60c0 con 0x7feac40c2900 2024-09-13T16:43:40.390 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac27fc640 1 -- 172.21.15.191:0/4210139000 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab0003520 con 0x7feac40c2900 2024-09-13T16:43:40.390 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac27fc640 1 -- 172.21.15.191:0/4210139000 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 491838158 0 0) 0x7feab0003520 con 0x7feac40c2900 2024-09-13T16:43:40.390 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac27fc640 1 -- 172.21.15.191:0/4210139000 >> v2:172.21.15.28:3300/0 conn(0x7feac40c6b10 msgr2=0x7feac4155b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:40.390 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac27fc640 1 --2- 172.21.15.191:0/4210139000 >> v2:172.21.15.28:3300/0 conn(0x7feac40c6b10 0x7feac4155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac27fc640 1 -- 172.21.15.191:0/4210139000 >> v1:172.21.15.28:6789/0 conn(0x7feac414f860 legacy=0x7feac40c5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:40.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac27fc640 1 -- 172.21.15.191:0/4210139000 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac40cb7b0 con 0x7feac40c2900 2024-09-13T16:43:40.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac27fc640 1 -- 172.21.15.191:0/4210139000 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1654382301 0 0) 0x7feab4001670 con 0x7feac40c2900 2024-09-13T16:43:40.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac27fc640 1 -- 172.21.15.191:0/4210139000 >> v1:172.21.15.191:6789/0 conn(0x7feac40c2900 legacy=0x7feac40c6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac27fc640 1 --2- 172.21.15.191:0/4210139000 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feab0003970 0x7feab0003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac27fc640 1 --2- 172.21.15.191:0/4210139000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feab00047c0 0x7feab0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac27fc640 1 --2- 172.21.15.191:0/4210139000 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feab00052a0 0x7feab000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac27fc640 1 -- 172.21.15.191:0/4210139000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac40cb4d0 con 0x7feab00047c0 2024-09-13T16:43:40.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac27fc640 1 -- 172.21.15.191:0/4210139000 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab000e3a0 con 0x7feab00052a0 2024-09-13T16:43:40.391 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac27fc640 1 -- 172.21.15.191:0/4210139000 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab000e560 con 0x7feab0003970 2024-09-13T16:43:40.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac37fe640 1 --2- 172.21.15.191:0/4210139000 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feab0003970 0x7feab0003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac3fff640 1 --2- 172.21.15.191:0/4210139000 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feab00052a0 0x7feab000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac2ffd640 1 --2- 172.21.15.191:0/4210139000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feab00047c0 0x7feab0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac37fe640 1 -- 172.21.15.191:0/4210139000 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feab00052a0 msgr2=0x7feab000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:40.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac37fe640 1 --2- 172.21.15.191:0/4210139000 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feab00052a0 0x7feab000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac37fe640 1 -- 172.21.15.191:0/4210139000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feab00047c0 msgr2=0x7feab0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:40.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac37fe640 1 --2- 172.21.15.191:0/4210139000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feab00047c0 0x7feab0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac37fe640 1 -- 172.21.15.191:0/4210139000 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab0003520 con 0x7feab0003970 2024-09-13T16:43:40.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac2ffd640 1 --2- 172.21.15.191:0/4210139000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feab00047c0 0x7feab0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:43:40.392 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.382+0000 7feac3fff640 1 -- 172.21.15.191:0/4210139000 reap_dead start 2024-09-13T16:43:40.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.383+0000 7feac37fe640 1 --2- 172.21.15.191:0/4210139000 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feab0003970 0x7feab0003d70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7feaac001620 tx=0x7feab00057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=399075230fb38376 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:40.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.383+0000 7feac27fc640 1 -- 172.21.15.191:0/4210139000 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac005270 con 0x7feab0003970 2024-09-13T16:43:40.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.383+0000 7feac27fc640 1 -- 172.21.15.191:0/4210139000 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac0053f0 con 0x7feab0003970 2024-09-13T16:43:40.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.383+0000 7feac27fc640 1 -- 172.21.15.191:0/4210139000 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac02a9d0 con 0x7feab0003970 2024-09-13T16:43:40.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.384+0000 7feac9fbc640 1 -- 172.21.15.191:0/4210139000 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feab0003970 msgr2=0x7feab0003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.384+0000 7feac9fbc640 1 --2- 172.21.15.191:0/4210139000 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feab0003970 0x7feab0003d70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7feaac001620 tx=0x7feab00057f0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.384+0000 7feac9fbc640 1 -- 172.21.15.191:0/4210139000 shutdown_connections 2024-09-13T16:43:40.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.384+0000 7feac9fbc640 1 --2- 172.21.15.191:0/4210139000 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feab0003970 0x7feab0003d70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.384+0000 7feac9fbc640 1 -- 172.21.15.191:0/4210139000 >> 172.21.15.191:0/4210139000 conn(0x7feac406d470 msgr2=0x7feac406af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:40.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.384+0000 7feac9fbc640 1 -- 172.21.15.191:0/4210139000 shutdown_connections 2024-09-13T16:43:40.393 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.384+0000 7feac9fbc640 1 -- 172.21.15.191:0/4210139000 wait complete. 2024-09-13T16:43:40.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.385+0000 7feac9fbc640 1 Processor -- start 2024-09-13T16:43:40.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.385+0000 7feac9fbc640 1 -- start start 2024-09-13T16:43:40.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac9fbc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feab0003970 0x7feac4154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac9fbc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac40cab50 0x7feac41ece50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac9fbc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feac40caf90 0x7feac41ed390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac9fbc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac414edf0 con 0x7feac40cab50 2024-09-13T16:43:40.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac9fbc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac414e200 con 0x7feab0003970 2024-09-13T16:43:40.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac9fbc640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac40c33a0 con 0x7feac40caf90 2024-09-13T16:43:40.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac3fff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feac40caf90 0x7feac41ed390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.394 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac37fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feab0003970 0x7feac4154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac37fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feab0003970 0x7feac4154b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59760/0 (socket says 172.21.15.191:59760) 2024-09-13T16:43:40.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac3fff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feac40caf90 0x7feac41ed390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:59782/0 (socket says 172.21.15.191:59782) 2024-09-13T16:43:40.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac37fe640 1 -- 172.21.15.191:0/2190784995 learned_addr learned my addr 172.21.15.191:0/2190784995 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-09-13T16:43:40.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac2ffd640 1 --2- 172.21.15.191:0/2190784995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac40cab50 0x7feac41ece50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac3fff640 1 -- 172.21.15.191:0/2190784995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feab0003970 msgr2=0x7feac4154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac3fff640 1 --2- 172.21.15.191:0/2190784995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feab0003970 0x7feac4154b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac3fff640 1 -- 172.21.15.191:0/2190784995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac40cab50 msgr2=0x7feac41ece50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac3fff640 1 --2- 172.21.15.191:0/2190784995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac40cab50 0x7feac41ece50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.395 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac3fff640 1 -- 172.21.15.191:0/2190784995 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac4155470 con 0x7feac40caf90 2024-09-13T16:43:40.396 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.386+0000 7feac3fff640 1 --2- 172.21.15.191:0/2190784995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feac40caf90 0x7feac41ed390 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7feac40c8f60 tx=0x7feabc00ede0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:40.396 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.387+0000 7feabbfff640 1 -- 172.21.15.191:0/2190784995 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc00cda0 con 0x7feac40caf90 2024-09-13T16:43:40.396 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.387+0000 7feabbfff640 1 -- 172.21.15.191:0/2190784995 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feabc003970 con 0x7feac40caf90 2024-09-13T16:43:40.396 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.387+0000 7feac9fbc640 1 -- 172.21.15.191:0/2190784995 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac414e450 con 0x7feac40caf90 2024-09-13T16:43:40.396 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.387+0000 7feabbfff640 1 -- 172.21.15.191:0/2190784995 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc003d20 con 0x7feac40caf90 2024-09-13T16:43:40.396 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.387+0000 7feac9fbc640 1 -- 172.21.15.191:0/2190784995 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac414ea10 con 0x7feac40caf90 2024-09-13T16:43:40.396 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.387+0000 7feabbfff640 1 -- 172.21.15.191:0/2190784995 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7feabc004180 con 0x7feac40caf90 2024-09-13T16:43:40.396 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.387+0000 7feabbfff640 1 -- 172.21.15.191:0/2190784995 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7feabc008030 con 0x7feac40caf90 2024-09-13T16:43:40.396 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.388+0000 7feac9fbc640 1 -- 172.21.15.191:0/2190784995 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac40679a0 con 0x7feac40caf90 2024-09-13T16:43:40.397 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.390+0000 7feabbfff640 1 -- 172.21.15.191:0/2190784995 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7feabc01c060 con 0x7feac40caf90 2024-09-13T16:43:40.423 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:40.422+0000 7f26d0638040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T16:43:40.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.458+0000 7feac9fbc640 1 -- 172.21.15.191:0/2190784995 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd new", "uuid": "67a0a7cd-ee9b-471c-a34f-d4fe97186aa0", "id": 1} v 0) v1 -- 0x7feac40c6400 con 0x7feac40caf90 2024-09-13T16:43:40.464 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.463+0000 7feabbfff640 1 -- 172.21.15.191:0/2190784995 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "67a0a7cd-ee9b-471c-a34f-d4fe97186aa0", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7feabc016430 con 0x7feac40caf90 2024-09-13T16:43:40.465 INFO:teuthology.orchestra.run.smithi191.stdout:1 2024-09-13T16:43:40.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.464+0000 7feac9fbc640 1 -- 172.21.15.191:0/2190784995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feac40caf90 msgr2=0x7feac41ed390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.464+0000 7feac9fbc640 1 --2- 172.21.15.191:0/2190784995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feac40caf90 0x7feac41ed390 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7feac40c8f60 tx=0x7feabc00ede0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.464+0000 7feac9fbc640 1 -- 172.21.15.191:0/2190784995 shutdown_connections 2024-09-13T16:43:40.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.464+0000 7feac9fbc640 1 --2- 172.21.15.191:0/2190784995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feac40caf90 0x7feac41ed390 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.464+0000 7feac9fbc640 1 --2- 172.21.15.191:0/2190784995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac40cab50 0x7feac41ece50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.464+0000 7feac9fbc640 1 --2- 172.21.15.191:0/2190784995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feab0003970 0x7feac4154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.464+0000 7feac9fbc640 1 -- 172.21.15.191:0/2190784995 >> 172.21.15.191:0/2190784995 conn(0x7feac406d470 msgr2=0x7feac41559d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:40.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.464+0000 7feac9fbc640 1 -- 172.21.15.191:0/2190784995 shutdown_connections 2024-09-13T16:43:40.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.464+0000 7feac9fbc640 1 -- 172.21.15.191:0/2190784995 wait complete. 2024-09-13T16:43:40.476 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new d3cd6efd-ff5f-4826-822c-79ebd46af2e3 2 2024-09-13T16:43:40.525 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:40.523+0000 7f98e4c38040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T16:43:40.573 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.571+0000 7f9e5b620640 1 Processor -- start 2024-09-13T16:43:40.573 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.572+0000 7f9e5b620640 1 -- start start 2024-09-13T16:43:40.574 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.572+0000 7f9e5b620640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e5414fa40 0x7f9e540c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.574 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.572+0000 7f9e5b620640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9e54071660 0x7f9e540c7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.574 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.572+0000 7f9e5b620640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9e540cb4c0 0x7f9e540cb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.575 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.572+0000 7f9e5b620640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e540c21c0 con 0x7f9e54071660 2024-09-13T16:43:40.575 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.572+0000 7f9e5b620640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e540722d0 con 0x7f9e540cb4c0 2024-09-13T16:43:40.575 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.572+0000 7f9e5b620640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e540c23f0 con 0x7f9e5414fa40 2024-09-13T16:43:40.575 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.572+0000 7f9e59bbd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9e540cb4c0 0x7f9e540cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.575 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.572+0000 7f9e593bc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e5414fa40 0x7f9e540c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.575 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.573+0000 7f9e593bc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e5414fa40 0x7f9e540c5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:59784/0 (socket says 172.21.15.191:59784) 2024-09-13T16:43:40.575 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.573+0000 7f9e59bbd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9e540cb4c0 0x7f9e540cb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59764/0 (socket says 172.21.15.191:59764) 2024-09-13T16:43:40.575 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.573+0000 7f9e593bc640 1 -- 172.21.15.191:0/2373454062 learned_addr learned my addr 172.21.15.191:0/2373454062 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-09-13T16:43:40.575 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.573+0000 7f9e58bbb640 1 --2- 172.21.15.191:0/2373454062 >> v2:172.21.15.28:3300/0 conn(0x7f9e54071660 0x7f9e540c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.575 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.573+0000 7f9e593bc640 1 -- 172.21.15.191:0/2373454062 >> v2:172.21.15.191:3300/0 conn(0x7f9e540cb4c0 msgr2=0x7f9e540cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:40.575 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.573+0000 7f9e593bc640 1 --2- 172.21.15.191:0/2373454062 >> v2:172.21.15.191:3300/0 conn(0x7f9e540cb4c0 0x7f9e540cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.575 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.573+0000 7f9e593bc640 1 -- 172.21.15.191:0/2373454062 >> v2:172.21.15.28:3300/0 conn(0x7f9e54071660 msgr2=0x7f9e540c7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:40.575 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.573+0000 7f9e593bc640 1 --2- 172.21.15.191:0/2373454062 >> v2:172.21.15.28:3300/0 conn(0x7f9e54071660 0x7f9e540c7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.576 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.573+0000 7f9e593bc640 1 -- 172.21.15.191:0/2373454062 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e54071d10 con 0x7f9e5414fa40 2024-09-13T16:43:40.576 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.573+0000 7f9e59bbd640 1 --2- 172.21.15.191:0/2373454062 >> v2:172.21.15.191:3300/0 conn(0x7f9e540cb4c0 0x7f9e540cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:40.576 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.573+0000 7f9e58bbb640 1 --2- 172.21.15.191:0/2373454062 >> v2:172.21.15.28:3300/0 conn(0x7f9e54071660 0x7f9e540c7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:43:40.576 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.573+0000 7f9e593bc640 1 --2- 172.21.15.191:0/2373454062 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e5414fa40 0x7f9e540c5f50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9e540c75b0 tx=0x7f9e4802c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=275debc2f0cb9b0c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:40.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.574+0000 7f9e47fff640 1 -- 172.21.15.191:0/2373454062 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e48032820 con 0x7f9e5414fa40 2024-09-13T16:43:40.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.574+0000 7f9e47fff640 1 -- 172.21.15.191:0/2373454062 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4802e040 con 0x7f9e5414fa40 2024-09-13T16:43:40.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.574+0000 7f9e47fff640 1 -- 172.21.15.191:0/2373454062 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e48032ba0 con 0x7f9e5414fa40 2024-09-13T16:43:40.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.574+0000 7f9e5b620640 1 -- 172.21.15.191:0/2373454062 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e5414fa40 msgr2=0x7f9e540c5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.574+0000 7f9e5b620640 1 --2- 172.21.15.191:0/2373454062 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e5414fa40 0x7f9e540c5f50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9e540c75b0 tx=0x7f9e4802c920 comp rx=0 tx=0).stop 2024-09-13T16:43:40.579 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.574+0000 7f9e5b620640 1 -- 172.21.15.191:0/2373454062 shutdown_connections 2024-09-13T16:43:40.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.574+0000 7f9e5b620640 1 --2- 172.21.15.191:0/2373454062 >> v2:172.21.15.191:3300/0 conn(0x7f9e540cb4c0 0x7f9e540cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.574+0000 7f9e5b620640 1 --2- 172.21.15.191:0/2373454062 >> v2:172.21.15.28:3300/0 conn(0x7f9e54071660 0x7f9e540c7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.574+0000 7f9e5b620640 1 --2- 172.21.15.191:0/2373454062 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e5414fa40 0x7f9e540c5f50 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.574+0000 7f9e5b620640 1 -- 172.21.15.191:0/2373454062 >> 172.21.15.191:0/2373454062 conn(0x7f9e5406d530 msgr2=0x7f9e5406af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:40.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.574+0000 7f9e5b620640 1 -- 172.21.15.191:0/2373454062 shutdown_connections 2024-09-13T16:43:40.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.574+0000 7f9e5b620640 1 -- 172.21.15.191:0/2373454062 wait complete. 2024-09-13T16:43:40.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.575+0000 7f9e5b620640 1 Processor -- start 2024-09-13T16:43:40.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.575+0000 7f9e5b620640 1 -- start start 2024-09-13T16:43:40.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.576+0000 7f9e5b620640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e54071660 0x7f9e5406a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.576+0000 7f9e5b620640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e540cb4c0 0x7f9e54143a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.576+0000 7f9e5b620640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e5414fa40 0x7f9e54141c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.580 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.576+0000 7f9e5b620640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5414ebb0 con 0x7f9e5414fa40 2024-09-13T16:43:40.581 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.576+0000 7f9e5b620640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e54147ad0 con 0x7f9e54071660 2024-09-13T16:43:40.581 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.576+0000 7f9e5b620640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e54147db0 con 0x7f9e540cb4c0 2024-09-13T16:43:40.581 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.576+0000 7f9e593bc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e54071660 0x7f9e5406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.581 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.576+0000 7f9e593bc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e54071660 0x7f9e5406a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59776/0 (socket says 172.21.15.191:59776) 2024-09-13T16:43:40.581 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.576+0000 7f9e593bc640 1 -- 172.21.15.191:0/4173383469 learned_addr learned my addr 172.21.15.191:0/4173383469 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-09-13T16:43:40.581 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.576+0000 7f9e58bbb640 1 --2- 172.21.15.191:0/4173383469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e540cb4c0 0x7f9e54143a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.581 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.576+0000 7f9e59bbd640 1 --2- 172.21.15.191:0/4173383469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e5414fa40 0x7f9e54141c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.581 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.576+0000 7f9e593bc640 1 -- 172.21.15.191:0/4173383469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e540cb4c0 msgr2=0x7f9e54143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.581 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.576+0000 7f9e593bc640 1 --2- 172.21.15.191:0/4173383469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e540cb4c0 0x7f9e54143a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.581 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.576+0000 7f9e593bc640 1 -- 172.21.15.191:0/4173383469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e5414fa40 msgr2=0x7f9e54141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.581 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.576+0000 7f9e593bc640 1 --2- 172.21.15.191:0/4173383469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e5414fa40 0x7f9e54141c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.581 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.576+0000 7f9e593bc640 1 -- 172.21.15.191:0/4173383469 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e4c009a70 con 0x7f9e54071660 2024-09-13T16:43:40.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.576+0000 7f9e59bbd640 1 --2- 172.21.15.191:0/4173383469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e5414fa40 0x7f9e54141c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:43:40.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.577+0000 7f9e593bc640 1 --2- 172.21.15.191:0/4173383469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e54071660 0x7f9e5406a540 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9e4802c960 tx=0x7f9e48004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:40.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.577+0000 7f9e45ffb640 1 -- 172.21.15.191:0/4173383469 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4803b7b0 con 0x7f9e54071660 2024-09-13T16:43:40.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.577+0000 7f9e45ffb640 1 -- 172.21.15.191:0/4173383469 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4803b910 con 0x7f9e54071660 2024-09-13T16:43:40.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.577+0000 7f9e45ffb640 1 -- 172.21.15.191:0/4173383469 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4803bca0 con 0x7f9e54071660 2024-09-13T16:43:40.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.577+0000 7f9e5b620640 1 -- 172.21.15.191:0/4173383469 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4800b770 con 0x7f9e54071660 2024-09-13T16:43:40.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.577+0000 7f9e5b620640 1 -- 172.21.15.191:0/4173383469 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5406a020 con 0x7f9e54071660 2024-09-13T16:43:40.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.577+0000 7f9e45ffb640 1 -- 172.21.15.191:0/4173383469 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f9e48045410 con 0x7f9e54071660 2024-09-13T16:43:40.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.577+0000 7f9e45ffb640 1 -- 172.21.15.191:0/4173383469 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f9e4803a7d0 con 0x7f9e54071660 2024-09-13T16:43:40.582 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.578+0000 7f9e5b620640 1 -- 172.21.15.191:0/4173383469 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e541482d0 con 0x7f9e54071660 2024-09-13T16:43:40.583 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:40.579+0000 7f98e4c38040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T16:43:40.583 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.580+0000 7f9e45ffb640 1 -- 172.21.15.191:0/4173383469 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f9e541482d0 con 0x7f9e54071660 2024-09-13T16:43:40.647 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.646+0000 7f9e5b620640 1 -- 172.21.15.191:0/4173383469 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d3cd6efd-ff5f-4826-822c-79ebd46af2e3", "id": 2} v 0) v1 -- 0x7f9e54069e80 con 0x7f9e54071660 2024-09-13T16:43:40.651 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.650+0000 7f9e45ffb640 1 -- 172.21.15.191:0/4173383469 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d3cd6efd-ff5f-4826-822c-79ebd46af2e3", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9e54069e80 con 0x7f9e54071660 2024-09-13T16:43:40.651 INFO:teuthology.orchestra.run.smithi191.stdout:2 2024-09-13T16:43:40.652 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.651+0000 7f9e5b620640 1 -- 172.21.15.191:0/4173383469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e54071660 msgr2=0x7f9e5406a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.652 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.651+0000 7f9e5b620640 1 --2- 172.21.15.191:0/4173383469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e54071660 0x7f9e5406a540 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9e4802c960 tx=0x7f9e48004730 comp rx=0 tx=0).stop 2024-09-13T16:43:40.652 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.651+0000 7f9e5b620640 1 -- 172.21.15.191:0/4173383469 shutdown_connections 2024-09-13T16:43:40.653 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.651+0000 7f9e5b620640 1 --2- 172.21.15.191:0/4173383469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e5414fa40 0x7f9e54141c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.653 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.651+0000 7f9e5b620640 1 --2- 172.21.15.191:0/4173383469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e540cb4c0 0x7f9e54143a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.653 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.651+0000 7f9e5b620640 1 --2- 172.21.15.191:0/4173383469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e54071660 0x7f9e5406a540 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.653 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.651+0000 7f9e5b620640 1 -- 172.21.15.191:0/4173383469 >> 172.21.15.191:0/4173383469 conn(0x7f9e5406d530 msgr2=0x7f9e540c5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:40.653 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.651+0000 7f9e5b620640 1 -- 172.21.15.191:0/4173383469 shutdown_connections 2024-09-13T16:43:40.653 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.651+0000 7f9e5b620640 1 -- 172.21.15.191:0/4173383469 wait complete. 2024-09-13T16:43:40.662 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 5f457889-9f7e-4799-8dc3-d58725734ce8 3 2024-09-13T16:43:40.734 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:40.733+0000 7f98e4c38040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T16:43:40.760 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.758+0000 7ff905a5a640 1 Processor -- start 2024-09-13T16:43:40.760 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.758+0000 7ff905a5a640 1 -- start start 2024-09-13T16:43:40.763 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.759+0000 7ff905a5a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff900152f30 0x7ff9000c3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.763 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.759+0000 7ff905a5a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9000ca7b0 0x7ff9000c3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.763 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.759+0000 7ff905a5a640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff900158a60 con 0x7ff9000c9520 2024-09-13T16:43:40.763 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.759+0000 7ff905a5a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9001599a0 con 0x7ff900152f30 2024-09-13T16:43:40.763 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.759+0000 7ff905a5a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff900158d20 con 0x7ff9000ca7b0 2024-09-13T16:43:40.763 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.759+0000 7ff8feffd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff900152f30 0x7ff9000c3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.763 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.759+0000 7ff8fe7fc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9000ca7b0 0x7ff9000c3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.764 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.759+0000 7ff8feffd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff900152f30 0x7ff9000c3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59780/0 (socket says 172.21.15.191:59780) 2024-09-13T16:43:40.764 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.759+0000 7ff8feffd640 1 -- 172.21.15.191:0/74467345 learned_addr learned my addr 172.21.15.191:0/74467345 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-09-13T16:43:40.764 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.760+0000 7ff8feffd640 1 -- 172.21.15.191:0/74467345 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9000ca7b0 msgr2=0x7ff9000c3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:40.764 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.760+0000 7ff8feffd640 1 --2- 172.21.15.191:0/74467345 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9000ca7b0 0x7ff9000c3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.764 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.760+0000 7ff8feffd640 1 -- 172.21.15.191:0/74467345 >> v1:172.21.15.28:6789/0 conn(0x7ff9000c9520 legacy=0x7ff90015c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:40.764 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.760+0000 7ff8feffd640 1 -- 172.21.15.191:0/74467345 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff900159000 con 0x7ff900152f30 2024-09-13T16:43:40.764 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.760+0000 7ff8fe7fc640 1 --2- 172.21.15.191:0/74467345 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9000ca7b0 0x7ff9000c3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:40.765 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.760+0000 7ff8feffd640 1 --2- 172.21.15.191:0/74467345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff900152f30 0x7ff9000c3540 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009870 tx=0x7ff8e802a200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e12f826be77c6eb1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:40.765 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.761+0000 7ff8fdffb640 1 -- 172.21.15.191:0/74467345 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8032440 con 0x7ff900152f30 2024-09-13T16:43:40.765 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.761+0000 7ff8fdffb640 1 -- 172.21.15.191:0/74467345 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e80325a0 con 0x7ff900152f30 2024-09-13T16:43:40.765 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.761+0000 7ff8fdffb640 1 -- 172.21.15.191:0/74467345 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8032930 con 0x7ff900152f30 2024-09-13T16:43:40.765 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.761+0000 7ff905a5a640 1 -- 172.21.15.191:0/74467345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff900152f30 msgr2=0x7ff9000c3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.765 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.761+0000 7ff905a5a640 1 --2- 172.21.15.191:0/74467345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff900152f30 0x7ff9000c3540 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009870 tx=0x7ff8e802a200 comp rx=0 tx=0).stop 2024-09-13T16:43:40.765 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.761+0000 7ff905a5a640 1 -- 172.21.15.191:0/74467345 shutdown_connections 2024-09-13T16:43:40.765 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.761+0000 7ff905a5a640 1 --2- 172.21.15.191:0/74467345 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9000ca7b0 0x7ff9000c3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.765 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.761+0000 7ff905a5a640 1 --2- 172.21.15.191:0/74467345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff900152f30 0x7ff9000c3540 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.765 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.761+0000 7ff905a5a640 1 -- 172.21.15.191:0/74467345 >> 172.21.15.191:0/74467345 conn(0x7ff900066070 msgr2=0x7ff900066480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:40.766 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.761+0000 7ff905a5a640 1 -- 172.21.15.191:0/74467345 shutdown_connections 2024-09-13T16:43:40.766 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.761+0000 7ff905a5a640 1 -- 172.21.15.191:0/74467345 wait complete. 2024-09-13T16:43:40.769 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.762+0000 7ff905a5a640 1 Processor -- start 2024-09-13T16:43:40.769 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.763+0000 7ff905a5a640 1 -- start start 2024-09-13T16:43:40.769 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.763+0000 7ff905a5a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9000c9520 0x7ff9001ef120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.769 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.763+0000 7ff905a5a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9000ca7b0 0x7ff90014fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.769 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.763+0000 7ff905a5a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff900152f30 0x7ff90014ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.769 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.763+0000 7ff905a5a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff900148dc0 con 0x7ff9000ca7b0 2024-09-13T16:43:40.770 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.763+0000 7ff905a5a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90015a060 con 0x7ff9000c9520 2024-09-13T16:43:40.770 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.763+0000 7ff905a5a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff90015a340 con 0x7ff900152f30 2024-09-13T16:43:40.770 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.763+0000 7ff8feffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9000c9520 0x7ff9001ef120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.770 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.763+0000 7ff8ff7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff900152f30 0x7ff90014ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.770 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.763+0000 7ff8feffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9000c9520 0x7ff9001ef120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59790/0 (socket says 172.21.15.191:59790) 2024-09-13T16:43:40.770 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.763+0000 7ff8feffd640 1 -- 172.21.15.191:0/3127011874 learned_addr learned my addr 172.21.15.191:0/3127011874 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-09-13T16:43:40.770 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.764+0000 7ff8fe7fc640 1 --2- 172.21.15.191:0/3127011874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9000ca7b0 0x7ff90014fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.770 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.764+0000 7ff8ff7fe640 1 -- 172.21.15.191:0/3127011874 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9000c9520 msgr2=0x7ff9001ef120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.770 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.764+0000 7ff8ff7fe640 1 --2- 172.21.15.191:0/3127011874 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9000c9520 0x7ff9001ef120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.770 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.764+0000 7ff8ff7fe640 1 -- 172.21.15.191:0/3127011874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9000ca7b0 msgr2=0x7ff90014fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.770 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.764+0000 7ff8ff7fe640 1 --2- 172.21.15.191:0/3127011874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9000ca7b0 0x7ff90014fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.770 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.764+0000 7ff8ff7fe640 1 -- 172.21.15.191:0/3127011874 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f0009400 con 0x7ff900152f30 2024-09-13T16:43:40.770 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.764+0000 7ff8feffd640 1 --2- 172.21.15.191:0/3127011874 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9000c9520 0x7ff9001ef120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:40.771 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.764+0000 7ff8fe7fc640 1 --2- 172.21.15.191:0/3127011874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9000ca7b0 0x7ff90014fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:43:40.771 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.764+0000 7ff8ff7fe640 1 --2- 172.21.15.191:0/3127011874 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff900152f30 0x7ff90014ed90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff8f800be00 tx=0x7ff8f800bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:40.771 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.764+0000 7ff8f7fff640 1 -- 172.21.15.191:0/3127011874 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f800ce70 con 0x7ff900152f30 2024-09-13T16:43:40.771 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.764+0000 7ff8f7fff640 1 -- 172.21.15.191:0/3127011874 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f8003970 con 0x7ff900152f30 2024-09-13T16:43:40.771 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.764+0000 7ff905a5a640 1 -- 172.21.15.191:0/3127011874 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e80094d0 con 0x7ff900152f30 2024-09-13T16:43:40.771 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.764+0000 7ff8f7fff640 1 -- 172.21.15.191:0/3127011874 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f8003d60 con 0x7ff900152f30 2024-09-13T16:43:40.771 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.764+0000 7ff905a5a640 1 -- 172.21.15.191:0/3127011874 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9001eec00 con 0x7ff900152f30 2024-09-13T16:43:40.771 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.764+0000 7ff8f7fff640 1 -- 172.21.15.191:0/3127011874 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7ff8f8003ec0 con 0x7ff900152f30 2024-09-13T16:43:40.771 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.764+0000 7ff8f7fff640 1 -- 172.21.15.191:0/3127011874 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7ff8f8014b00 con 0x7ff900152f30 2024-09-13T16:43:40.772 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.765+0000 7ff905a5a640 1 -- 172.21.15.191:0/3127011874 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9001eee20 con 0x7ff900152f30 2024-09-13T16:43:40.772 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.769+0000 7ff8f7fff640 1 -- 172.21.15.191:0/3127011874 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7ff8f8022060 con 0x7ff900152f30 2024-09-13T16:43:40.805 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:40.803+0000 7f26d0638040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T16:43:40.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.836+0000 7ff905a5a640 1 -- 172.21.15.191:0/3127011874 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5f457889-9f7e-4799-8dc3-d58725734ce8", "id": 3} v 0) v1 -- 0x7ff9000b2390 con 0x7ff900152f30 2024-09-13T16:43:40.841 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.840+0000 7ff8f7fff640 1 -- 172.21.15.191:0/3127011874 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5f457889-9f7e-4799-8dc3-d58725734ce8", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff8f802b0b0 con 0x7ff900152f30 2024-09-13T16:43:40.841 INFO:teuthology.orchestra.run.smithi191.stdout:3 2024-09-13T16:43:40.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.841+0000 7ff905a5a640 1 -- 172.21.15.191:0/3127011874 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff900152f30 msgr2=0x7ff90014ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.841+0000 7ff905a5a640 1 --2- 172.21.15.191:0/3127011874 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff900152f30 0x7ff90014ed90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff8f800be00 tx=0x7ff8f800bef0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.841+0000 7ff905a5a640 1 -- 172.21.15.191:0/3127011874 shutdown_connections 2024-09-13T16:43:40.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.841+0000 7ff905a5a640 1 --2- 172.21.15.191:0/3127011874 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff900152f30 0x7ff90014ed90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.841+0000 7ff905a5a640 1 --2- 172.21.15.191:0/3127011874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9000ca7b0 0x7ff90014fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.841+0000 7ff905a5a640 1 --2- 172.21.15.191:0/3127011874 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9000c9520 0x7ff9001ef120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.841+0000 7ff905a5a640 1 -- 172.21.15.191:0/3127011874 >> 172.21.15.191:0/3127011874 conn(0x7ff900066070 msgr2=0x7ff9000c4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:40.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.841+0000 7ff905a5a640 1 -- 172.21.15.191:0/3127011874 shutdown_connections 2024-09-13T16:43:40.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.841+0000 7ff905a5a640 1 -- 172.21.15.191:0/3127011874 wait complete. 2024-09-13T16:43:40.853 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 5cc0d9dd-9486-4893-b94a-7dfdf32edbee 4 2024-09-13T16:43:40.885 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:40.883+0000 7f26d0638040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T16:43:40.950 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.949+0000 7f58c8f65640 1 Processor -- start 2024-09-13T16:43:40.951 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.949+0000 7f58c8f65640 1 -- start start 2024-09-13T16:43:40.952 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.950+0000 7f58c8f65640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f58c414f860 0x7f58c40c2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.952 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.950+0000 7f58c8f65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58c40c5370 0x7f58c40c6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.952 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.950+0000 7f58c8f65640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f58c40c6940 0x7f58c4155b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.952 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.950+0000 7f58c8f65640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c40c2e40 con 0x7f58c40c6940 2024-09-13T16:43:40.952 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.950+0000 7f58c8f65640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c4155680 con 0x7f58c414f860 2024-09-13T16:43:40.952 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.950+0000 7f58c8f65640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c40c60c0 con 0x7f58c40c5370 2024-09-13T16:43:40.952 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.950+0000 7f58c1d9b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58c40c5370 0x7f58c40c6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.953 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.950+0000 7f58c259c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f58c414f860 0x7f58c40c2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.953 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.950+0000 7f58c1d9b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58c40c5370 0x7f58c40c6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:59836/0 (socket says 172.21.15.191:59836) 2024-09-13T16:43:40.953 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.950+0000 7f58c259c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f58c414f860 0x7f58c40c2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59804/0 (socket says 172.21.15.191:59804) 2024-09-13T16:43:40.953 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.950+0000 7f58c1d9b640 1 -- 172.21.15.191:0/1303607457 learned_addr learned my addr 172.21.15.191:0/1303607457 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-09-13T16:43:40.953 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.951+0000 7f58c2d9d640 1 --2- 172.21.15.191:0/1303607457 >> v2:172.21.15.28:3300/0 conn(0x7f58c40c6940 0x7f58c4155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.953 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.951+0000 7f58c1d9b640 1 -- 172.21.15.191:0/1303607457 >> v2:172.21.15.191:3300/0 conn(0x7f58c414f860 msgr2=0x7f58c40c2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:40.953 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.951+0000 7f58c1d9b640 1 --2- 172.21.15.191:0/1303607457 >> v2:172.21.15.191:3300/0 conn(0x7f58c414f860 0x7f58c40c2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.953 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.951+0000 7f58c1d9b640 1 -- 172.21.15.191:0/1303607457 >> v2:172.21.15.28:3300/0 conn(0x7f58c40c6940 msgr2=0x7f58c4155b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:40.953 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.951+0000 7f58c1d9b640 1 --2- 172.21.15.191:0/1303607457 >> v2:172.21.15.28:3300/0 conn(0x7f58c40c6940 0x7f58c4155b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.953 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.951+0000 7f58c1d9b640 1 -- 172.21.15.191:0/1303607457 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c40cb4d0 con 0x7f58c40c5370 2024-09-13T16:43:40.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.951+0000 7f58c1d9b640 1 --2- 172.21.15.191:0/1303607457 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58c40c5370 0x7f58c40c6400 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f58c406b9e0 tx=0x7f58ac02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8b6e6935cde99bd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:40.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.951+0000 7f58c159a640 1 -- 172.21.15.191:0/1303607457 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac032440 con 0x7f58c40c5370 2024-09-13T16:43:40.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.952+0000 7f58c159a640 1 -- 172.21.15.191:0/1303607457 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ac0325a0 con 0x7f58c40c5370 2024-09-13T16:43:40.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.952+0000 7f58c159a640 1 -- 172.21.15.191:0/1303607457 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac032930 con 0x7f58c40c5370 2024-09-13T16:43:40.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.952+0000 7f58c8f65640 1 -- 172.21.15.191:0/1303607457 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58c40c5370 msgr2=0x7f58c40c6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.952+0000 7f58c8f65640 1 --2- 172.21.15.191:0/1303607457 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58c40c5370 0x7f58c40c6400 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f58c406b9e0 tx=0x7f58ac02a270 comp rx=0 tx=0).stop 2024-09-13T16:43:40.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.952+0000 7f58c8f65640 1 -- 172.21.15.191:0/1303607457 shutdown_connections 2024-09-13T16:43:40.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.952+0000 7f58c8f65640 1 --2- 172.21.15.191:0/1303607457 >> v2:172.21.15.28:3300/0 conn(0x7f58c40c6940 0x7f58c4155b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.952+0000 7f58c8f65640 1 --2- 172.21.15.191:0/1303607457 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58c40c5370 0x7f58c40c6400 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.952+0000 7f58c8f65640 1 --2- 172.21.15.191:0/1303607457 >> v2:172.21.15.191:3300/0 conn(0x7f58c414f860 0x7f58c40c2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.952+0000 7f58c8f65640 1 -- 172.21.15.191:0/1303607457 >> 172.21.15.191:0/1303607457 conn(0x7f58c406d470 msgr2=0x7f58c406af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:40.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.952+0000 7f58c8f65640 1 -- 172.21.15.191:0/1303607457 shutdown_connections 2024-09-13T16:43:40.956 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.952+0000 7f58c8f65640 1 -- 172.21.15.191:0/1303607457 wait complete. 2024-09-13T16:43:40.956 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.954+0000 7f58c8f65640 1 Processor -- start 2024-09-13T16:43:40.956 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.954+0000 7f58c8f65640 1 -- start start 2024-09-13T16:43:40.958 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.954+0000 7f58c8f65640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c40c5370 0x7f58c414d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.958 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.954+0000 7f58c8f65640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58c40c6940 0x7f58c40ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.958 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.954+0000 7f58c8f65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58c414f860 0x7f58c41532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:40.958 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.954+0000 7f58c8f65640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c41461d0 con 0x7f58c40c5370 2024-09-13T16:43:40.958 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.954+0000 7f58c8f65640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c414e420 con 0x7f58c40c6940 2024-09-13T16:43:40.958 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.954+0000 7f58c8f65640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c414e670 con 0x7f58c414f860 2024-09-13T16:43:40.958 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.954+0000 7f58c1d9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58c40c6940 0x7f58c40ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.958 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.954+0000 7f58c2d9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58c414f860 0x7f58c41532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.958 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.954+0000 7f58c1d9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58c40c6940 0x7f58c40ca4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59808/0 (socket says 172.21.15.191:59808) 2024-09-13T16:43:40.958 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.954+0000 7f58c1d9b640 1 -- 172.21.15.191:0/2129050613 learned_addr learned my addr 172.21.15.191:0/2129050613 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-09-13T16:43:40.959 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.954+0000 7f58c259c640 1 --2- 172.21.15.191:0/2129050613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c40c5370 0x7f58c414d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:40.959 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.955+0000 7f58c1d9b640 1 -- 172.21.15.191:0/2129050613 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58c414f860 msgr2=0x7f58c41532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.959 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.955+0000 7f58c1d9b640 1 --2- 172.21.15.191:0/2129050613 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58c414f860 0x7f58c41532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.959 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.955+0000 7f58c1d9b640 1 -- 172.21.15.191:0/2129050613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c40c5370 msgr2=0x7f58c414d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:40.959 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.955+0000 7f58c1d9b640 1 --2- 172.21.15.191:0/2129050613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c40c5370 0x7f58c414d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:40.959 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.955+0000 7f58c1d9b640 1 -- 172.21.15.191:0/2129050613 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac0094d0 con 0x7f58c40c6940 2024-09-13T16:43:40.959 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.955+0000 7f58c259c640 1 --2- 172.21.15.191:0/2129050613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c40c5370 0x7f58c414d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:43:40.959 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.955+0000 7f58c2d9d640 1 --2- 172.21.15.191:0/2129050613 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58c414f860 0x7f58c41532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:40.960 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.955+0000 7f58c1d9b640 1 --2- 172.21.15.191:0/2129050613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58c40c6940 0x7f58c40ca4c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f58ac000c00 tx=0x7f58ac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:40.960 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.955+0000 7f58baffd640 1 -- 172.21.15.191:0/2129050613 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac032440 con 0x7f58c40c6940 2024-09-13T16:43:40.960 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.955+0000 7f58baffd640 1 -- 172.21.15.191:0/2129050613 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ac030980 con 0x7f58c40c6940 2024-09-13T16:43:40.960 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.955+0000 7f58c8f65640 1 -- 172.21.15.191:0/2129050613 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c4154540 con 0x7f58c40c6940 2024-09-13T16:43:40.960 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.955+0000 7f58baffd640 1 -- 172.21.15.191:0/2129050613 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac030d10 con 0x7f58c40c6940 2024-09-13T16:43:40.960 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.955+0000 7f58c8f65640 1 -- 172.21.15.191:0/2129050613 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c4154b00 con 0x7f58c40c6940 2024-09-13T16:43:40.960 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.955+0000 7f58baffd640 1 -- 172.21.15.191:0/2129050613 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f58ac039040 con 0x7f58c40c6940 2024-09-13T16:43:40.960 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.955+0000 7f58baffd640 1 -- 172.21.15.191:0/2129050613 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f58ac03b910 con 0x7f58c40c6940 2024-09-13T16:43:40.960 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.956+0000 7f58c8f65640 1 -- 172.21.15.191:0/2129050613 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c40c4330 con 0x7f58c40c6940 2024-09-13T16:43:40.960 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:40.958+0000 7f58baffd640 1 -- 172.21.15.191:0/2129050613 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f58ac066e30 con 0x7f58c40c6940 2024-09-13T16:43:41.027 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.026+0000 7f58c8f65640 1 -- 172.21.15.191:0/2129050613 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5cc0d9dd-9486-4893-b94a-7dfdf32edbee", "id": 4} v 0) v1 -- 0x7f58c414fc40 con 0x7f58c40c6940 2024-09-13T16:43:41.031 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.030+0000 7f58baffd640 1 -- 172.21.15.191:0/2129050613 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5cc0d9dd-9486-4893-b94a-7dfdf32edbee", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f58ac055070 con 0x7f58c40c6940 2024-09-13T16:43:41.031 INFO:teuthology.orchestra.run.smithi191.stdout:4 2024-09-13T16:43:41.032 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.031+0000 7f58c8f65640 1 -- 172.21.15.191:0/2129050613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58c40c6940 msgr2=0x7f58c40ca4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.032 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.031+0000 7f58c8f65640 1 --2- 172.21.15.191:0/2129050613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58c40c6940 0x7f58c40ca4c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f58ac000c00 tx=0x7f58ac0046c0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.032 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.031+0000 7f58c8f65640 1 -- 172.21.15.191:0/2129050613 shutdown_connections 2024-09-13T16:43:41.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.031+0000 7f58c8f65640 1 --2- 172.21.15.191:0/2129050613 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f58c414f860 0x7f58c41532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.031+0000 7f58c8f65640 1 --2- 172.21.15.191:0/2129050613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f58c40c6940 0x7f58c40ca4c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.031+0000 7f58c8f65640 1 --2- 172.21.15.191:0/2129050613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c40c5370 0x7f58c414d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.031+0000 7f58c8f65640 1 -- 172.21.15.191:0/2129050613 >> 172.21.15.191:0/2129050613 conn(0x7f58c406d470 msgr2=0x7f58c41562d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:41.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.031+0000 7f58c8f65640 1 -- 172.21.15.191:0/2129050613 shutdown_connections 2024-09-13T16:43:41.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.031+0000 7f58c8f65640 1 -- 172.21.15.191:0/2129050613 wait complete. 2024-09-13T16:43:41.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b 5 2024-09-13T16:43:41.061 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:41.059+0000 7f26d0638040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T16:43:41.141 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.140+0000 7ffa1863a640 1 Processor -- start 2024-09-13T16:43:41.141 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.140+0000 7ffa1863a640 1 -- start start 2024-09-13T16:43:41.142 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.141+0000 7ffa1863a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ffa100c4df0 0x7ffa1006a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.142 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.141+0000 7ffa1863a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffa1006ab50 0x7ffa10150700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.142 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.141+0000 7ffa1863a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1006f2e0 con 0x7ffa100c5e70 2024-09-13T16:43:41.143 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.141+0000 7ffa1863a640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa100c6300 con 0x7ffa1006ab50 2024-09-13T16:43:41.143 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.141+0000 7ffa1863a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa100c65e0 con 0x7ffa100c4df0 2024-09-13T16:43:41.143 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.141+0000 7ffa163d6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ffa100c4df0 0x7ffa1006a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.143 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.141+0000 7ffa163d6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ffa100c4df0 0x7ffa1006a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59816/0 (socket says 172.21.15.191:59816) 2024-09-13T16:43:41.143 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.141+0000 7ffa163d6640 1 -- 172.21.15.191:0/1264634817 learned_addr learned my addr 172.21.15.191:0/1264634817 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-09-13T16:43:41.143 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.141+0000 7ffa16bd7640 1 --2- 172.21.15.191:0/1264634817 >> v2:172.21.15.28:3300/0 conn(0x7ffa1006ab50 0x7ffa10150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.143 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.141+0000 7ffa163d6640 1 -- 172.21.15.191:0/1264634817 >> v2:172.21.15.28:3300/0 conn(0x7ffa1006ab50 msgr2=0x7ffa10150700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:41.143 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.141+0000 7ffa163d6640 1 --2- 172.21.15.191:0/1264634817 >> v2:172.21.15.28:3300/0 conn(0x7ffa1006ab50 0x7ffa10150700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.143 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.141+0000 7ffa163d6640 1 -- 172.21.15.191:0/1264634817 >> v1:172.21.15.191:6789/0 conn(0x7ffa100c5e70 legacy=0x7ffa10156ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.143 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.141+0000 7ffa163d6640 1 -- 172.21.15.191:0/1264634817 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1006e9b0 con 0x7ffa100c4df0 2024-09-13T16:43:41.145 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.142+0000 7ffa163d6640 1 --2- 172.21.15.191:0/1264634817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa100c4df0 0x7ffa1006a610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ffa1006bd70 tx=0x7ffa0c02a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=324e33f08f2fcbeb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:41.145 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.142+0000 7ffa153d4640 1 -- 172.21.15.191:0/1264634817 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c032440 con 0x7ffa100c4df0 2024-09-13T16:43:41.145 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.142+0000 7ffa153d4640 1 -- 172.21.15.191:0/1264634817 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0c0325a0 con 0x7ffa100c4df0 2024-09-13T16:43:41.145 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.142+0000 7ffa153d4640 1 -- 172.21.15.191:0/1264634817 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c032930 con 0x7ffa100c4df0 2024-09-13T16:43:41.145 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.142+0000 7ffa1863a640 1 -- 172.21.15.191:0/1264634817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa100c4df0 msgr2=0x7ffa1006a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.145 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.142+0000 7ffa1863a640 1 --2- 172.21.15.191:0/1264634817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa100c4df0 0x7ffa1006a610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ffa1006bd70 tx=0x7ffa0c02a1e0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.145 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.142+0000 7ffa1863a640 1 -- 172.21.15.191:0/1264634817 shutdown_connections 2024-09-13T16:43:41.145 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.142+0000 7ffa1863a640 1 --2- 172.21.15.191:0/1264634817 >> v2:172.21.15.28:3300/0 conn(0x7ffa1006ab50 0x7ffa10150700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.145 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.142+0000 7ffa1863a640 1 --2- 172.21.15.191:0/1264634817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa100c4df0 0x7ffa1006a610 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.145 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.142+0000 7ffa1863a640 1 -- 172.21.15.191:0/1264634817 >> 172.21.15.191:0/1264634817 conn(0x7ffa10066280 msgr2=0x7ffa1006af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:41.145 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.142+0000 7ffa1863a640 1 -- 172.21.15.191:0/1264634817 shutdown_connections 2024-09-13T16:43:41.145 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.142+0000 7ffa1863a640 1 -- 172.21.15.191:0/1264634817 wait complete. 2024-09-13T16:43:41.146 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.144+0000 7ffa1863a640 1 Processor -- start 2024-09-13T16:43:41.146 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.144+0000 7ffa1863a640 1 -- start start 2024-09-13T16:43:41.148 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.144+0000 7ffa1863a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1006ab50 0x7ffa10150740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.148 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.144+0000 7ffa1863a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa100c4df0 0x7ffa101584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.148 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.144+0000 7ffa1863a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa10158a30 0x7ffa10148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.148 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.144+0000 7ffa1863a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1015ada0 con 0x7ffa1006ab50 2024-09-13T16:43:41.148 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.144+0000 7ffa1863a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa10154210 con 0x7ffa100c4df0 2024-09-13T16:43:41.148 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.144+0000 7ffa1863a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa101efc70 con 0x7ffa10158a30 2024-09-13T16:43:41.148 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.144+0000 7ffa15bd5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa100c4df0 0x7ffa101584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.149 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.144+0000 7ffa15bd5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa100c4df0 0x7ffa101584f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59828/0 (socket says 172.21.15.191:59828) 2024-09-13T16:43:41.149 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.144+0000 7ffa15bd5640 1 -- 172.21.15.191:0/4140122054 learned_addr learned my addr 172.21.15.191:0/4140122054 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-09-13T16:43:41.149 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.144+0000 7ffa16bd7640 1 --2- 172.21.15.191:0/4140122054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa10158a30 0x7ffa10148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.149 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.144+0000 7ffa163d6640 1 --2- 172.21.15.191:0/4140122054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1006ab50 0x7ffa10150740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.149 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.145+0000 7ffa15bd5640 1 -- 172.21.15.191:0/4140122054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa10158a30 msgr2=0x7ffa10148db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.149 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.145+0000 7ffa15bd5640 1 --2- 172.21.15.191:0/4140122054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa10158a30 0x7ffa10148db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.149 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.145+0000 7ffa15bd5640 1 -- 172.21.15.191:0/4140122054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1006ab50 msgr2=0x7ffa10150740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.149 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.145+0000 7ffa15bd5640 1 --2- 172.21.15.191:0/4140122054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1006ab50 0x7ffa10150740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.149 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.145+0000 7ffa15bd5640 1 -- 172.21.15.191:0/4140122054 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0c0094d0 con 0x7ffa100c4df0 2024-09-13T16:43:41.149 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.145+0000 7ffa16bd7640 1 --2- 172.21.15.191:0/4140122054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa10158a30 0x7ffa10148db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:41.149 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.145+0000 7ffa163d6640 1 --2- 172.21.15.191:0/4140122054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1006ab50 0x7ffa10150740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:43:41.149 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.145+0000 7ffa15bd5640 1 --2- 172.21.15.191:0/4140122054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa100c4df0 0x7ffa101584f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ffa0000c400 tx=0x7ffa0000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:41.149 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.145+0000 7ffa06ffd640 1 -- 172.21.15.191:0/4140122054 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa000052d0 con 0x7ffa100c4df0 2024-09-13T16:43:41.150 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.145+0000 7ffa06ffd640 1 -- 172.21.15.191:0/4140122054 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa00005430 con 0x7ffa100c4df0 2024-09-13T16:43:41.150 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.145+0000 7ffa06ffd640 1 -- 172.21.15.191:0/4140122054 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa00003830 con 0x7ffa100c4df0 2024-09-13T16:43:41.150 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.145+0000 7ffa1863a640 1 -- 172.21.15.191:0/4140122054 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa101f01a0 con 0x7ffa100c4df0 2024-09-13T16:43:41.150 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.145+0000 7ffa1863a640 1 -- 172.21.15.191:0/4140122054 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa10158ee0 con 0x7ffa100c4df0 2024-09-13T16:43:41.150 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.145+0000 7ffa06ffd640 1 -- 172.21.15.191:0/4140122054 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7ffa00003c90 con 0x7ffa100c4df0 2024-09-13T16:43:41.150 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.145+0000 7ffa06ffd640 1 -- 172.21.15.191:0/4140122054 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7ffa000129a0 con 0x7ffa100c4df0 2024-09-13T16:43:41.152 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.146+0000 7ffa1863a640 1 -- 172.21.15.191:0/4140122054 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa10067460 con 0x7ffa100c4df0 2024-09-13T16:43:41.152 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.148+0000 7ffa06ffd640 1 -- 172.21.15.191:0/4140122054 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7ffa00017070 con 0x7ffa100c4df0 2024-09-13T16:43:41.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.215+0000 7ffa1863a640 1 -- 172.21.15.191:0/4140122054 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b", "id": 5} v 0) v1 -- 0x7ffa100c6b60 con 0x7ffa100c4df0 2024-09-13T16:43:41.219 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.218+0000 7ffa06ffd640 1 -- 172.21.15.191:0/4140122054 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7ffa00012420 con 0x7ffa100c4df0 2024-09-13T16:43:41.220 INFO:teuthology.orchestra.run.smithi191.stdout:5 2024-09-13T16:43:41.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.219+0000 7ffa1863a640 1 -- 172.21.15.191:0/4140122054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa100c4df0 msgr2=0x7ffa101584f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.219+0000 7ffa1863a640 1 --2- 172.21.15.191:0/4140122054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa100c4df0 0x7ffa101584f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ffa0000c400 tx=0x7ffa0000c8c0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.219+0000 7ffa1863a640 1 -- 172.21.15.191:0/4140122054 shutdown_connections 2024-09-13T16:43:41.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.219+0000 7ffa1863a640 1 --2- 172.21.15.191:0/4140122054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffa10158a30 0x7ffa10148db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.219+0000 7ffa1863a640 1 --2- 172.21.15.191:0/4140122054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffa100c4df0 0x7ffa101584f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.219+0000 7ffa1863a640 1 --2- 172.21.15.191:0/4140122054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1006ab50 0x7ffa10150740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.219+0000 7ffa1863a640 1 -- 172.21.15.191:0/4140122054 >> 172.21.15.191:0/4140122054 conn(0x7ffa10066280 msgr2=0x7ffa100cbe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:41.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.219+0000 7ffa1863a640 1 -- 172.21.15.191:0/4140122054 shutdown_connections 2024-09-13T16:43:41.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.219+0000 7ffa1863a640 1 -- 172.21.15.191:0/4140122054 wait complete. 2024-09-13T16:43:41.231 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 53a405e2-64d2-4baa-9a8c-16b03368865c 6 2024-09-13T16:43:41.329 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.327+0000 7f3b0c11f640 1 Processor -- start 2024-09-13T16:43:41.329 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.328+0000 7f3b0c11f640 1 -- start start 2024-09-13T16:43:41.330 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.328+0000 7f3b0c11f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3b04150ab0 0x7f3b040b0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.330 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.328+0000 7f3b0c11f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3b04065f70 0x7f3b04156a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.330 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.328+0000 7f3b0c11f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b04066390 con 0x7f3b04156fa0 2024-09-13T16:43:41.330 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.328+0000 7f3b0c11f640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b040b10c0 con 0x7f3b04150ab0 2024-09-13T16:43:41.330 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.328+0000 7f3b0c11f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b04070630 con 0x7f3b04065f70 2024-09-13T16:43:41.331 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.328+0000 7f3b096ba640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3b04065f70 0x7f3b04156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.331 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.329+0000 7f3b096ba640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3b04065f70 0x7f3b04156a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59834/0 (socket says 172.21.15.191:59834) 2024-09-13T16:43:41.331 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.329+0000 7f3b096ba640 1 -- 172.21.15.191:0/1936926401 learned_addr learned my addr 172.21.15.191:0/1936926401 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-09-13T16:43:41.331 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.329+0000 7f3b09ebb640 1 --2- 172.21.15.191:0/1936926401 >> v2:172.21.15.28:3300/0 conn(0x7f3b04150ab0 0x7f3b040b0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.331 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.329+0000 7f3b096ba640 1 -- 172.21.15.191:0/1936926401 >> v2:172.21.15.28:3300/0 conn(0x7f3b04150ab0 msgr2=0x7f3b040b0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:41.331 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.329+0000 7f3b096ba640 1 --2- 172.21.15.191:0/1936926401 >> v2:172.21.15.28:3300/0 conn(0x7f3b04150ab0 0x7f3b040b0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.331 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.329+0000 7f3b096ba640 1 -- 172.21.15.191:0/1936926401 >> v1:172.21.15.191:6789/0 conn(0x7f3b04156fa0 legacy=0x7f3b040c6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.331 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.329+0000 7f3b096ba640 1 -- 172.21.15.191:0/1936926401 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b040b0d90 con 0x7f3b04065f70 2024-09-13T16:43:41.332 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.330+0000 7f3b096ba640 1 --2- 172.21.15.191:0/1936926401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b04065f70 0x7f3b04156a60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3b0415af00 tx=0x7f3af402a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f978d1d2f04f7206 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:41.332 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.330+0000 7f3b08eb9640 1 -- 172.21.15.191:0/1936926401 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af4032640 con 0x7f3b04065f70 2024-09-13T16:43:41.332 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.330+0000 7f3b08eb9640 1 -- 172.21.15.191:0/1936926401 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af40327a0 con 0x7f3b04065f70 2024-09-13T16:43:41.332 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.330+0000 7f3b08eb9640 1 -- 172.21.15.191:0/1936926401 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af4032b30 con 0x7f3b04065f70 2024-09-13T16:43:41.333 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.331+0000 7f3b0c11f640 1 -- 172.21.15.191:0/1936926401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b04065f70 msgr2=0x7f3b04156a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.333 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.331+0000 7f3b0c11f640 1 --2- 172.21.15.191:0/1936926401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b04065f70 0x7f3b04156a60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3b0415af00 tx=0x7f3af402a300 comp rx=0 tx=0).stop 2024-09-13T16:43:41.333 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.331+0000 7f3b0c11f640 1 -- 172.21.15.191:0/1936926401 shutdown_connections 2024-09-13T16:43:41.333 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.331+0000 7f3b0c11f640 1 --2- 172.21.15.191:0/1936926401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b04065f70 0x7f3b04156a60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.333 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.331+0000 7f3b0c11f640 1 --2- 172.21.15.191:0/1936926401 >> v2:172.21.15.28:3300/0 conn(0x7f3b04150ab0 0x7f3b040b0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.333 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.331+0000 7f3b0c11f640 1 -- 172.21.15.191:0/1936926401 >> 172.21.15.191:0/1936926401 conn(0x7f3b0406c7c0 msgr2=0x7f3b0406cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:41.333 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.331+0000 7f3b0c11f640 1 -- 172.21.15.191:0/1936926401 shutdown_connections 2024-09-13T16:43:41.333 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.331+0000 7f3b0c11f640 1 -- 172.21.15.191:0/1936926401 wait complete. 2024-09-13T16:43:41.335 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.332+0000 7f3b0c11f640 1 Processor -- start 2024-09-13T16:43:41.336 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.332+0000 7f3b0c11f640 1 -- start start 2024-09-13T16:43:41.336 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b0c11f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b04065f70 0x7f3b041e0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.336 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b0c11f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04150ab0 0x7f3b041e1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.336 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b0c11f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b04156fa0 0x7f3b041efcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.336 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b0c11f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b041dfa30 con 0x7f3b04150ab0 2024-09-13T16:43:41.336 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b0c11f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b041dfd10 con 0x7f3b04156fa0 2024-09-13T16:43:41.336 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b0c11f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b041dfff0 con 0x7f3b04065f70 2024-09-13T16:43:41.336 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b09ebb640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b04065f70 0x7f3b041e0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.336 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b09ebb640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b04065f70 0x7f3b041e0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:59864/0 (socket says 172.21.15.191:59864) 2024-09-13T16:43:41.336 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b0a6bc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b04156fa0 0x7f3b041efcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.336 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b09ebb640 1 -- 172.21.15.191:0/1684616076 learned_addr learned my addr 172.21.15.191:0/1684616076 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-09-13T16:43:41.336 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b096ba640 1 --2- 172.21.15.191:0/1684616076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04150ab0 0x7f3b041e1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.337 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b0a6bc640 1 -- 172.21.15.191:0/1684616076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b04065f70 msgr2=0x7f3b041e0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.337 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b0a6bc640 1 --2- 172.21.15.191:0/1684616076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b04065f70 0x7f3b041e0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.337 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b0a6bc640 1 -- 172.21.15.191:0/1684616076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04150ab0 msgr2=0x7f3b041e1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.337 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b0a6bc640 1 --2- 172.21.15.191:0/1684616076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04150ab0 0x7f3b041e1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.337 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b0a6bc640 1 -- 172.21.15.191:0/1684616076 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af40094d0 con 0x7f3b04156fa0 2024-09-13T16:43:41.337 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b09ebb640 1 --2- 172.21.15.191:0/1684616076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b04065f70 0x7f3b041e0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:41.337 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b0a6bc640 1 --2- 172.21.15.191:0/1684616076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b04156fa0 0x7f3b041efcf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3afc00c700 tx=0x7f3afc00cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:41.337 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3afa7fc640 1 -- 172.21.15.191:0/1684616076 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc003b70 con 0x7f3b04156fa0 2024-09-13T16:43:41.337 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3afa7fc640 1 -- 172.21.15.191:0/1684616076 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3afc003cd0 con 0x7f3b04156fa0 2024-09-13T16:43:41.337 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b0c11f640 1 -- 172.21.15.191:0/1684616076 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b041e02d0 con 0x7f3b04156fa0 2024-09-13T16:43:41.337 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3afa7fc640 1 -- 172.21.15.191:0/1684616076 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc005310 con 0x7f3b04156fa0 2024-09-13T16:43:41.337 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.333+0000 7f3b0c11f640 1 -- 172.21.15.191:0/1684616076 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b041df6f0 con 0x7f3b04156fa0 2024-09-13T16:43:41.337 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.334+0000 7f3afa7fc640 1 -- 172.21.15.191:0/1684616076 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f3afc00a640 con 0x7f3b04156fa0 2024-09-13T16:43:41.338 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.334+0000 7f3afa7fc640 1 -- 172.21.15.191:0/1684616076 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f3afc012a10 con 0x7f3b04156fa0 2024-09-13T16:43:41.338 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.334+0000 7f3b0c11f640 1 -- 172.21.15.191:0/1684616076 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b041e07f0 con 0x7f3b04156fa0 2024-09-13T16:43:41.338 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.336+0000 7f3afa7fc640 1 -- 172.21.15.191:0/1684616076 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f3b041e07f0 con 0x7f3b04156fa0 2024-09-13T16:43:41.404 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.403+0000 7f3b0c11f640 1 -- 172.21.15.191:0/1684616076 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "53a405e2-64d2-4baa-9a8c-16b03368865c", "id": 6} v 0) v1 -- 0x7f3b04157690 con 0x7f3b04156fa0 2024-09-13T16:43:41.408 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.407+0000 7f3afa7fc640 1 -- 172.21.15.191:0/1684616076 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "53a405e2-64d2-4baa-9a8c-16b03368865c", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3b04066390 con 0x7f3b04156fa0 2024-09-13T16:43:41.408 INFO:teuthology.orchestra.run.smithi191.stdout:6 2024-09-13T16:43:41.409 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.408+0000 7f3b0c11f640 1 -- 172.21.15.191:0/1684616076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b04156fa0 msgr2=0x7f3b041efcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.409 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.408+0000 7f3b0c11f640 1 --2- 172.21.15.191:0/1684616076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b04156fa0 0x7f3b041efcf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3afc00c700 tx=0x7f3afc00cbc0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.409 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.408+0000 7f3b0c11f640 1 -- 172.21.15.191:0/1684616076 shutdown_connections 2024-09-13T16:43:41.410 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.408+0000 7f3b0c11f640 1 --2- 172.21.15.191:0/1684616076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b04156fa0 0x7f3b041efcf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.410 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.408+0000 7f3b0c11f640 1 --2- 172.21.15.191:0/1684616076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b04150ab0 0x7f3b041e1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.410 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.408+0000 7f3b0c11f640 1 --2- 172.21.15.191:0/1684616076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b04065f70 0x7f3b041e0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.410 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.408+0000 7f3b0c11f640 1 -- 172.21.15.191:0/1684616076 >> 172.21.15.191:0/1684616076 conn(0x7f3b0406c7c0 msgr2=0x7f3b04154d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:41.410 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.408+0000 7f3b0c11f640 1 -- 172.21.15.191:0/1684616076 shutdown_connections 2024-09-13T16:43:41.410 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.408+0000 7f3b0c11f640 1 -- 172.21.15.191:0/1684616076 wait complete. 2024-09-13T16:43:41.420 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 001391e2-0bf7-4397-9a21-f44efea6b269 7 2024-09-13T16:43:41.518 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.517+0000 7fc759261640 1 Processor -- start 2024-09-13T16:43:41.519 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.517+0000 7fc759261640 1 -- start start 2024-09-13T16:43:41.521 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.518+0000 7fc759261640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc7540c94f0 0x7fc7541575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.521 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.518+0000 7fc759261640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc7540c66c0 0x7fc7540b00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.522 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.518+0000 7fc759261640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc754157b10 con 0x7fc7540b2100 2024-09-13T16:43:41.522 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.518+0000 7fc759261640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc75406ee20 con 0x7fc7540c66c0 2024-09-13T16:43:41.522 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.518+0000 7fc759261640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7540b2680 con 0x7fc7540c94f0 2024-09-13T16:43:41.522 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.518+0000 7fc752ffd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc7540c94f0 0x7fc7541575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.522 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.518+0000 7fc752ffd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc7540c94f0 0x7fc7541575d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59840/0 (socket says 172.21.15.191:59840) 2024-09-13T16:43:41.522 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.518+0000 7fc752ffd640 1 -- 172.21.15.191:0/2956249722 learned_addr learned my addr 172.21.15.191:0/2956249722 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-09-13T16:43:41.522 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.518+0000 7fc7527fc640 1 --2- 172.21.15.191:0/2956249722 >> v2:172.21.15.28:3300/0 conn(0x7fc7540c66c0 0x7fc7540b00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.522 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.518+0000 7fc752ffd640 1 -- 172.21.15.191:0/2956249722 >> v2:172.21.15.28:3300/0 conn(0x7fc7540c66c0 msgr2=0x7fc7540b00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:41.522 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.518+0000 7fc752ffd640 1 --2- 172.21.15.191:0/2956249722 >> v2:172.21.15.28:3300/0 conn(0x7fc7540c66c0 0x7fc7540b00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.522 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.518+0000 7fc752ffd640 1 -- 172.21.15.191:0/2956249722 >> v1:172.21.15.28:6789/0 conn(0x7fc7540b2100 legacy=0x7fc754157fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:41.523 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.518+0000 7fc752ffd640 1 -- 172.21.15.191:0/2956249722 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75406e250 con 0x7fc7540c94f0 2024-09-13T16:43:41.523 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.519+0000 7fc752ffd640 1 --2- 172.21.15.191:0/2956249722 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7540c94f0 0x7fc7541575d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc754067b00 tx=0x7fc73c02a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fbf89f310f691a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:41.523 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.519+0000 7fc751ffb640 1 -- 172.21.15.191:0/2956249722 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c032440 con 0x7fc7540c94f0 2024-09-13T16:43:41.523 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.519+0000 7fc751ffb640 1 -- 172.21.15.191:0/2956249722 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73c0325a0 con 0x7fc7540c94f0 2024-09-13T16:43:41.523 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.519+0000 7fc751ffb640 1 -- 172.21.15.191:0/2956249722 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c032930 con 0x7fc7540c94f0 2024-09-13T16:43:41.523 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.520+0000 7fc759261640 1 -- 172.21.15.191:0/2956249722 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7540c94f0 msgr2=0x7fc7541575d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.523 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.520+0000 7fc759261640 1 --2- 172.21.15.191:0/2956249722 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7540c94f0 0x7fc7541575d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc754067b00 tx=0x7fc73c02a230 comp rx=0 tx=0).stop 2024-09-13T16:43:41.523 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.520+0000 7fc759261640 1 -- 172.21.15.191:0/2956249722 shutdown_connections 2024-09-13T16:43:41.523 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.520+0000 7fc759261640 1 --2- 172.21.15.191:0/2956249722 >> v2:172.21.15.28:3300/0 conn(0x7fc7540c66c0 0x7fc7540b00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.523 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.520+0000 7fc759261640 1 --2- 172.21.15.191:0/2956249722 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7540c94f0 0x7fc7541575d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.523 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.520+0000 7fc759261640 1 -- 172.21.15.191:0/2956249722 >> 172.21.15.191:0/2956249722 conn(0x7fc75406fb50 msgr2=0x7fc75406ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:41.524 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.520+0000 7fc759261640 1 -- 172.21.15.191:0/2956249722 shutdown_connections 2024-09-13T16:43:41.524 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.520+0000 7fc759261640 1 -- 172.21.15.191:0/2956249722 wait complete. 2024-09-13T16:43:41.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.521+0000 7fc759261640 1 Processor -- start 2024-09-13T16:43:41.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.521+0000 7fc759261640 1 -- start start 2024-09-13T16:43:41.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc759261640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7540b2100 0x7fc754155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc759261640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7540c66c0 0x7fc7540ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc759261640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7540c94f0 0x7fc7540ae150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc759261640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7540ad060 con 0x7fc7540b2100 2024-09-13T16:43:41.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc759261640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7540acdc0 con 0x7fc7540c66c0 2024-09-13T16:43:41.529 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc759261640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7541ef320 con 0x7fc7540c94f0 2024-09-13T16:43:41.529 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc7537fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7540c94f0 0x7fc7540ae150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.529 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc7527fc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7540c66c0 0x7fc7540ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.529 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc7527fc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7540c66c0 0x7fc7540ca4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59842/0 (socket says 172.21.15.191:59842) 2024-09-13T16:43:41.529 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc7537fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7540c94f0 0x7fc7540ae150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:59876/0 (socket says 172.21.15.191:59876) 2024-09-13T16:43:41.530 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc7527fc640 1 -- 172.21.15.191:0/316013976 learned_addr learned my addr 172.21.15.191:0/316013976 (peer_addr_for_me v2:172.21.15.191:0/0) 2024-09-13T16:43:41.530 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc752ffd640 1 --2- 172.21.15.191:0/316013976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7540b2100 0x7fc754155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.530 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc7537fe640 1 -- 172.21.15.191:0/316013976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7540c66c0 msgr2=0x7fc7540ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc7537fe640 1 --2- 172.21.15.191:0/316013976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7540c66c0 0x7fc7540ca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc7537fe640 1 -- 172.21.15.191:0/316013976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7540b2100 msgr2=0x7fc754155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc7537fe640 1 --2- 172.21.15.191:0/316013976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7540b2100 0x7fc754155a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc7537fe640 1 -- 172.21.15.191:0/316013976 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73c0094d0 con 0x7fc7540c94f0 2024-09-13T16:43:41.531 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc7537fe640 1 --2- 172.21.15.191:0/316013976 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7540c94f0 0x7fc7540ae150 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc74c00be00 tx=0x7fc74c00bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:41.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.522+0000 7fc74b7fe640 1 -- 172.21.15.191:0/316013976 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c00ce70 con 0x7fc7540c94f0 2024-09-13T16:43:41.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.523+0000 7fc74b7fe640 1 -- 172.21.15.191:0/316013976 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74c003970 con 0x7fc7540c94f0 2024-09-13T16:43:41.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.523+0000 7fc74b7fe640 1 -- 172.21.15.191:0/316013976 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c003d60 con 0x7fc7540c94f0 2024-09-13T16:43:41.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.523+0000 7fc759261640 1 -- 172.21.15.191:0/316013976 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7541ef600 con 0x7fc7540c94f0 2024-09-13T16:43:41.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.523+0000 7fc759261640 1 -- 172.21.15.191:0/316013976 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc75406cf10 con 0x7fc7540c94f0 2024-09-13T16:43:41.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.523+0000 7fc74b7fe640 1 -- 172.21.15.191:0/316013976 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fc74c003ec0 con 0x7fc7540c94f0 2024-09-13T16:43:41.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.523+0000 7fc74b7fe640 1 -- 172.21.15.191:0/316013976 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fc74c014da0 con 0x7fc7540c94f0 2024-09-13T16:43:41.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.524+0000 7fc759261640 1 -- 172.21.15.191:0/316013976 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7540ade70 con 0x7fc7540c94f0 2024-09-13T16:43:41.533 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.526+0000 7fc74b7fe640 1 -- 172.21.15.191:0/316013976 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fc7540ade70 con 0x7fc7540c94f0 2024-09-13T16:43:41.594 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.593+0000 7fc759261640 1 -- 172.21.15.191:0/316013976 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd new", "uuid": "001391e2-0bf7-4397-9a21-f44efea6b269", "id": 7} v 0) v1 -- 0x7fc7541575d0 con 0x7fc7540c94f0 2024-09-13T16:43:41.597 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.596+0000 7fc74b7fe640 1 -- 172.21.15.191:0/316013976 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "001391e2-0bf7-4397-9a21-f44efea6b269", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc74c022060 con 0x7fc7540c94f0 2024-09-13T16:43:41.598 INFO:teuthology.orchestra.run.smithi191.stdout:7 2024-09-13T16:43:41.599 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.597+0000 7fc759261640 1 -- 172.21.15.191:0/316013976 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7540c94f0 msgr2=0x7fc7540ae150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.599 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.597+0000 7fc759261640 1 --2- 172.21.15.191:0/316013976 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7540c94f0 0x7fc7540ae150 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc74c00be00 tx=0x7fc74c00bef0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.599 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.597+0000 7fc759261640 1 -- 172.21.15.191:0/316013976 shutdown_connections 2024-09-13T16:43:41.599 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.597+0000 7fc759261640 1 --2- 172.21.15.191:0/316013976 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7540c94f0 0x7fc7540ae150 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.599 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.597+0000 7fc759261640 1 --2- 172.21.15.191:0/316013976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7540c66c0 0x7fc7540ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.600 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.597+0000 7fc759261640 1 --2- 172.21.15.191:0/316013976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7540b2100 0x7fc754155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.600 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.597+0000 7fc759261640 1 -- 172.21.15.191:0/316013976 >> 172.21.15.191:0/316013976 conn(0x7fc75406fb50 msgr2=0x7fc7540c58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:41.600 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.597+0000 7fc759261640 1 -- 172.21.15.191:0/316013976 shutdown_connections 2024-09-13T16:43:41.600 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-13T16:43:41.597+0000 7fc759261640 1 -- 172.21.15.191:0/316013976 wait complete. 2024-09-13T16:43:41.609 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-13T16:43:41.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-13T16:43:41.613 INFO:tasks.ceph.osd.0:Started 2024-09-13T16:43:41.613 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-13T16:43:41.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-13T16:43:41.617 INFO:tasks.ceph.osd.1:Started 2024-09-13T16:43:41.617 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-13T16:43:41.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-13T16:43:41.621 INFO:tasks.ceph.osd.2:Started 2024-09-13T16:43:41.621 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-13T16:43:41.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-13T16:43:41.625 INFO:tasks.ceph.osd.3:Started 2024-09-13T16:43:41.625 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-13T16:43:41.625 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-13T16:43:41.627 INFO:tasks.ceph.osd.4:Started 2024-09-13T16:43:41.628 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-13T16:43:41.628 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-13T16:43:41.630 INFO:tasks.ceph.osd.5:Started 2024-09-13T16:43:41.630 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-13T16:43:41.630 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-13T16:43:41.632 INFO:tasks.ceph.osd.6:Started 2024-09-13T16:43:41.632 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-13T16:43:41.633 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-13T16:43:41.635 INFO:tasks.ceph.osd.7:Started 2024-09-13T16:43:41.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:43:41.698 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:41.697+0000 7f98e4c38040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T16:43:41.760 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:41.759+0000 7f98e4c38040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T16:43:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.838+0000 7fc735000640 1 Processor -- start 2024-09-13T16:43:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.838+0000 7fc735000640 1 -- start start 2024-09-13T16:43:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.839+0000 7fc735000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc730152da0 0x7fc7300c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.839+0000 7fc735000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc73006ac90 0x7fc73006b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.839+0000 7fc735000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73015cda0 con 0x7fc7300ca410 2024-09-13T16:43:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.839+0000 7fc735000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7300c39e0 con 0x7fc73006ac90 2024-09-13T16:43:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.839+0000 7fc735000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc73006b680 con 0x7fc730152da0 2024-09-13T16:43:41.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.840+0000 7fc72e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc73006ac90 0x7fc73006b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.840+0000 7fc72d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc7300ca410 0x7fc730065fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56392/0 (socket says 172.21.15.28:56392) 2024-09-13T16:43:41.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.840+0000 7fc72d400640 1 -- 172.21.15.28:0/1647909373 learned_addr learned my addr 172.21.15.28:0/1647909373 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:43:41.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.840+0000 7fc72de00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc730152da0 0x7fc7300c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.840+0000 7fc72de00640 1 -- 172.21.15.28:0/1647909373 >> v2:172.21.15.28:3300/0 conn(0x7fc73006ac90 msgr2=0x7fc73006b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:41.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.840+0000 7fc72de00640 1 --2- 172.21.15.28:0/1647909373 >> v2:172.21.15.28:3300/0 conn(0x7fc73006ac90 0x7fc73006b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.840+0000 7fc72ca00640 1 -- 172.21.15.28:0/1647909373 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512705914 0 0) 0x7fc73015cda0 con 0x7fc7300ca410 2024-09-13T16:43:41.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.840+0000 7fc72de00640 1 -- 172.21.15.28:0/1647909373 >> v1:172.21.15.28:6789/0 conn(0x7fc7300ca410 legacy=0x7fc730065fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.840+0000 7fc72de00640 1 -- 172.21.15.28:0/1647909373 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7300c3be0 con 0x7fc730152da0 2024-09-13T16:43:41.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.841+0000 7fc72de00640 1 --2- 172.21.15.28:0/1647909373 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc730152da0 0x7fc7300c34a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc7200097a0 tx=0x7fc72002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=144740d452c5d161 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:41.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.842+0000 7fc72ca00640 1 -- 172.21.15.28:0/1647909373 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc720032440 con 0x7fc730152da0 2024-09-13T16:43:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.842+0000 7fc72ca00640 1 -- 172.21.15.28:0/1647909373 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7200325a0 con 0x7fc730152da0 2024-09-13T16:43:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.842+0000 7fc72ca00640 1 -- 172.21.15.28:0/1647909373 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc720032930 con 0x7fc730152da0 2024-09-13T16:43:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.842+0000 7fc735000640 1 -- 172.21.15.28:0/1647909373 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc730152da0 msgr2=0x7fc7300c34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.842+0000 7fc735000640 1 --2- 172.21.15.28:0/1647909373 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc730152da0 0x7fc7300c34a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc7200097a0 tx=0x7fc72002a210 comp rx=0 tx=0).stop 2024-09-13T16:43:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.842+0000 7fc735000640 1 -- 172.21.15.28:0/1647909373 shutdown_connections 2024-09-13T16:43:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.842+0000 7fc735000640 1 --2- 172.21.15.28:0/1647909373 >> v2:172.21.15.28:3300/0 conn(0x7fc73006ac90 0x7fc73006b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.842+0000 7fc735000640 1 --2- 172.21.15.28:0/1647909373 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc730152da0 0x7fc7300c34a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.843+0000 7fc735000640 1 -- 172.21.15.28:0/1647909373 >> 172.21.15.28:0/1647909373 conn(0x7fc7300ce810 msgr2=0x7fc7300cec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.843+0000 7fc735000640 1 -- 172.21.15.28:0/1647909373 shutdown_connections 2024-09-13T16:43:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.843+0000 7fc735000640 1 -- 172.21.15.28:0/1647909373 wait complete. 2024-09-13T16:43:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.844+0000 7fc735000640 1 Processor -- start 2024-09-13T16:43:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.844+0000 7fc735000640 1 -- start start 2024-09-13T16:43:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.845+0000 7fc735000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc73006ac90 0x7fc7300c7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.845+0000 7fc735000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7300ca410 0x7fc7300c7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.845+0000 7fc735000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc730152da0 0x7fc730148f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.845+0000 7fc735000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7300c6fc0 con 0x7fc7300ca410 2024-09-13T16:43:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.845+0000 7fc735000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc730155700 con 0x7fc730152da0 2024-09-13T16:43:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.845+0000 7fc735000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7301ef770 con 0x7fc73006ac90 2024-09-13T16:43:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.845+0000 7fc72de00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc73006ac90 0x7fc7300c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.845+0000 7fc72d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7300ca410 0x7fc7300c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.845+0000 7fc72de00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc73006ac90 0x7fc7300c7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:35978/0 (socket says 172.21.15.28:35978) 2024-09-13T16:43:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.845+0000 7fc72de00640 1 -- 172.21.15.28:0/1003012783 learned_addr learned my addr 172.21.15.28:0/1003012783 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.846+0000 7fc72e800640 1 --2- 172.21.15.28:0/1003012783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc730152da0 0x7fc730148f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.846+0000 7fc72de00640 1 -- 172.21.15.28:0/1003012783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc730152da0 msgr2=0x7fc730148f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.846+0000 7fc72de00640 1 --2- 172.21.15.28:0/1003012783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc730152da0 0x7fc730148f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.846+0000 7fc72de00640 1 -- 172.21.15.28:0/1003012783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7300ca410 msgr2=0x7fc7300c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.846+0000 7fc72de00640 1 --2- 172.21.15.28:0/1003012783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7300ca410 0x7fc7300c7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.846+0000 7fc72de00640 1 -- 172.21.15.28:0/1003012783 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc720009400 con 0x7fc73006ac90 2024-09-13T16:43:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.846+0000 7fc72d400640 1 --2- 172.21.15.28:0/1003012783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7300ca410 0x7fc7300c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.846+0000 7fc72de00640 1 --2- 172.21.15.28:0/1003012783 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc73006ac90 0x7fc7300c7450 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc7200098c0 tx=0x7fc7200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.846+0000 7fc726000640 1 -- 172.21.15.28:0/1003012783 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc720030b80 con 0x7fc73006ac90 2024-09-13T16:43:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.847+0000 7fc726000640 1 -- 172.21.15.28:0/1003012783 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc720032e90 con 0x7fc73006ac90 2024-09-13T16:43:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.847+0000 7fc735000640 1 -- 172.21.15.28:0/1003012783 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7301eebf0 con 0x7fc73006ac90 2024-09-13T16:43:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.847+0000 7fc726000640 1 -- 172.21.15.28:0/1003012783 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc720006e10 con 0x7fc73006ac90 2024-09-13T16:43:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.847+0000 7fc735000640 1 -- 172.21.15.28:0/1003012783 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7301ef1b0 con 0x7fc73006ac90 2024-09-13T16:43:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.847+0000 7fc726000640 1 -- 172.21.15.28:0/1003012783 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fc720039040 con 0x7fc73006ac90 2024-09-13T16:43:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.847+0000 7fc726000640 1 -- 172.21.15.28:0/1003012783 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc72003bad0 con 0x7fc73006ac90 2024-09-13T16:43:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.848+0000 7fc735000640 1 -- 172.21.15.28:0/1003012783 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7300cdab0 con 0x7fc73006ac90 2024-09-13T16:43:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.850+0000 7fc726000640 1 -- 172.21.15.28:0/1003012783 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fc72004c060 con 0x7fc73006ac90 2024-09-13T16:43:41.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.912+0000 7fc735000640 1 -- 172.21.15.28:0/1003012783 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7300cacf0 con 0x7fc73006ac90 2024-09-13T16:43:41.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.913+0000 7fc726000640 1 -- 172.21.15.28:0/1003012783 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fc720042070 con 0x7fc73006ac90 2024-09-13T16:43:41.914 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:41.914 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:43:41.595256+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:43:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.914+0000 7fc735000640 1 -- 172.21.15.28:0/1003012783 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc73006ac90 msgr2=0x7fc7300c7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.914+0000 7fc735000640 1 --2- 172.21.15.28:0/1003012783 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc73006ac90 0x7fc7300c7450 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc7200098c0 tx=0x7fc7200046c0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.914+0000 7fc735000640 1 -- 172.21.15.28:0/1003012783 shutdown_connections 2024-09-13T16:43:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.914+0000 7fc735000640 1 --2- 172.21.15.28:0/1003012783 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc730152da0 0x7fc730148f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.914+0000 7fc735000640 1 --2- 172.21.15.28:0/1003012783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7300ca410 0x7fc7300c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.914+0000 7fc735000640 1 --2- 172.21.15.28:0/1003012783 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc73006ac90 0x7fc7300c7450 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.914+0000 7fc735000640 1 -- 172.21.15.28:0/1003012783 >> 172.21.15.28:0/1003012783 conn(0x7fc7300ce810 msgr2=0x7fc7300ccd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.914+0000 7fc735000640 1 -- 172.21.15.28:0/1003012783 shutdown_connections 2024-09-13T16:43:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:41.914+0000 7fc735000640 1 -- 172.21.15.28:0/1003012783 wait complete. 2024-09-13T16:43:41.930 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-13T16:43:41.930 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-13T16:43:42.127 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:42.126+0000 7f98e4c38040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T16:43:42.209 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:42.208+0000 7f98e4c38040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T16:43:42.227 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-09-13T16:43:42.226+0000 7f13425bf640 -1 Falling back to public interface 2024-09-13T16:43:42.237 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-09-13T16:43:42.235+0000 7eff2eada640 -1 Falling back to public interface 2024-09-13T16:43:42.243 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-09-13T16:43:42.242+0000 7f0416e9c640 -1 Falling back to public interface 2024-09-13T16:43:42.254 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-13T16:43:42.253+0000 7f4b60ae6640 -1 Falling back to public interface 2024-09-13T16:43:42.301 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-13T16:43:42.300+0000 7fec16f23640 -1 Falling back to public interface 2024-09-13T16:43:42.303 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:42.302+0000 7f26d0638040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T16:43:42.321 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:42.320+0000 7f98e4c38040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T16:43:42.321 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-13T16:43:42.320+0000 7f32afc20640 -1 Falling back to public interface 2024-09-13T16:43:42.340 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-13T16:43:42.338+0000 7fc105302640 -1 Falling back to public interface 2024-09-13T16:43:42.355 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-09-13T16:43:42.353+0000 7fcdf940e640 -1 Falling back to public interface 2024-09-13T16:43:42.366 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:42.365+0000 7f98e4c38040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T16:43:42.370 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:42.369+0000 7f26d0638040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T16:43:42.476 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:42.475+0000 7f98e4c38040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T16:43:42.531 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:42.530+0000 7f98e4c38040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T16:43:42.576 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:42.574+0000 7f98e4c38040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T16:43:42.685 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:42.684+0000 7f98e4c38040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T16:43:42.734 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:42.733+0000 7f98e4c38040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T16:43:42.737 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T16:43:42.786 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:42.785+0000 7f98e4c38040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T16:43:42.830 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:42.829+0000 7f26d0638040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T16:43:42.854 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:42.852+0000 7f98e4c38040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T16:43:42.874 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.872+0000 7f2803e00640 1 Processor -- start 2024-09-13T16:43:42.874 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.872+0000 7f2803e00640 1 -- start start 2024-09-13T16:43:42.877 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.874+0000 7f2803e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f27fc06dbb0 0x7f27fc071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:42.877 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.874+0000 7f2803e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f27fc06fee0 0x7f27fc14edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:42.877 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.874+0000 7f2803e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27fc06df90 0x7f27fc14f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:42.877 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.874+0000 7f2803e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27fc0723a0 con 0x7f27fc06dbb0 2024-09-13T16:43:42.877 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.874+0000 7f2803e00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27fc14f840 con 0x7f27fc06fee0 2024-09-13T16:43:42.877 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.874+0000 7f2803e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc070e40 con 0x7f27fc06df90 2024-09-13T16:43:42.877 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.875+0000 7f2800a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f27fc06fee0 0x7f27fc14edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:42.878 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.875+0000 7f2801400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f27fc06dbb0 0x7f27fc071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:42.878 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.875+0000 7f2800a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f27fc06fee0 0x7f27fc14edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:38558/0 (socket says 172.21.15.28:38558) 2024-09-13T16:43:42.878 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.875+0000 7f2800a00640 1 -- 172.21.15.28:0/3214943012 learned_addr learned my addr 172.21.15.28:0/3214943012 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:42.878 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.875+0000 7f2801e00640 1 --2- 172.21.15.28:0/3214943012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27fc06df90 0x7f27fc14f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:42.878 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.875+0000 7f2800a00640 1 -- 172.21.15.28:0/3214943012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27fc06df90 msgr2=0x7f27fc14f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:42.878 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.875+0000 7f2800a00640 1 --2- 172.21.15.28:0/3214943012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27fc06df90 0x7f27fc14f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:42.878 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.875+0000 7f2800a00640 1 -- 172.21.15.28:0/3214943012 >> v2:172.21.15.28:3300/0 conn(0x7f27fc06dbb0 msgr2=0x7f27fc071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:42.879 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.875+0000 7f2800a00640 1 --2- 172.21.15.28:0/3214943012 >> v2:172.21.15.28:3300/0 conn(0x7f27fc06dbb0 0x7f27fc071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:42.879 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.875+0000 7f2800a00640 1 -- 172.21.15.28:0/3214943012 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc1587d0 con 0x7f27fc06fee0 2024-09-13T16:43:42.879 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.875+0000 7f2801e00640 1 --2- 172.21.15.28:0/3214943012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27fc06df90 0x7f27fc14f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:42.882 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.876+0000 7f2800a00640 1 --2- 172.21.15.28:0/3214943012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27fc06fee0 0x7f27fc14edc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f27fc0c1820 tx=0x7f27e802a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9636cffe7849c0b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:42.882 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.876+0000 7f27fbe00640 1 -- 172.21.15.28:0/3214943012 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e8032440 con 0x7f27fc06fee0 2024-09-13T16:43:42.882 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.876+0000 7f27fbe00640 1 -- 172.21.15.28:0/3214943012 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e80325a0 con 0x7f27fc06fee0 2024-09-13T16:43:42.882 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.876+0000 7f27fbe00640 1 -- 172.21.15.28:0/3214943012 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e8032930 con 0x7f27fc06fee0 2024-09-13T16:43:42.882 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.877+0000 7f2803e00640 1 -- 172.21.15.28:0/3214943012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27fc06fee0 msgr2=0x7f27fc14edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:42.882 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.877+0000 7f2803e00640 1 --2- 172.21.15.28:0/3214943012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27fc06fee0 0x7f27fc14edc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f27fc0c1820 tx=0x7f27e802a210 comp rx=0 tx=0).stop 2024-09-13T16:43:42.883 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.877+0000 7f2803e00640 1 -- 172.21.15.28:0/3214943012 shutdown_connections 2024-09-13T16:43:42.883 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.877+0000 7f2803e00640 1 --2- 172.21.15.28:0/3214943012 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27fc06df90 0x7f27fc14f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:42.883 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.877+0000 7f2803e00640 1 --2- 172.21.15.28:0/3214943012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27fc06fee0 0x7f27fc14edc0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:42.883 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.877+0000 7f2803e00640 1 --2- 172.21.15.28:0/3214943012 >> v2:172.21.15.28:3300/0 conn(0x7f27fc06dbb0 0x7f27fc071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:42.883 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.877+0000 7f2803e00640 1 -- 172.21.15.28:0/3214943012 >> 172.21.15.28:0/3214943012 conn(0x7f27fc069e70 msgr2=0x7f27fc06a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:42.883 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.877+0000 7f2803e00640 1 -- 172.21.15.28:0/3214943012 shutdown_connections 2024-09-13T16:43:42.883 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.877+0000 7f2803e00640 1 -- 172.21.15.28:0/3214943012 wait complete. 2024-09-13T16:43:42.883 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.879+0000 7f2803e00640 1 Processor -- start 2024-09-13T16:43:42.883 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.879+0000 7f2803e00640 1 -- start start 2024-09-13T16:43:42.883 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.880+0000 7f2803e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27fc06dbb0 0x7f27fc1ecc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:42.883 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.880+0000 7f2803e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27fc06df90 0x7f27fc14ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:42.883 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.880+0000 7f2803e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27fc06fee0 0x7f27fc0c8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:42.884 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.880+0000 7f2803e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc0c99c0 con 0x7f27fc06fee0 2024-09-13T16:43:42.884 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.880+0000 7f2803e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc143a80 con 0x7f27fc06dbb0 2024-09-13T16:43:42.884 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.880+0000 7f2803e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc143cd0 con 0x7f27fc06df90 2024-09-13T16:43:42.884 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.880+0000 7f2801400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27fc06dbb0 0x7f27fc1ecc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:42.884 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.880+0000 7f2800a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27fc06df90 0x7f27fc14ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:42.884 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.880+0000 7f2801400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27fc06dbb0 0x7f27fc1ecc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:38562/0 (socket says 172.21.15.28:38562) 2024-09-13T16:43:42.884 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.880+0000 7f2801400640 1 -- 172.21.15.28:0/711344127 learned_addr learned my addr 172.21.15.28:0/711344127 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:42.884 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.880+0000 7f2800a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27fc06df90 0x7f27fc14ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:36000/0 (socket says 172.21.15.28:36000) 2024-09-13T16:43:42.884 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.880+0000 7f2801e00640 1 --2- 172.21.15.28:0/711344127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27fc06fee0 0x7f27fc0c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:42.884 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.880+0000 7f2800a00640 1 -- 172.21.15.28:0/711344127 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27fc06dbb0 msgr2=0x7f27fc1ecc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:42.884 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.880+0000 7f2800a00640 1 --2- 172.21.15.28:0/711344127 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27fc06dbb0 0x7f27fc1ecc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:42.884 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.881+0000 7f2800a00640 1 -- 172.21.15.28:0/711344127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27fc06fee0 msgr2=0x7f27fc0c8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:42.884 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.881+0000 7f2800a00640 1 --2- 172.21.15.28:0/711344127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27fc06fee0 0x7f27fc0c8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:42.885 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.881+0000 7f2800a00640 1 -- 172.21.15.28:0/711344127 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f4009a70 con 0x7f27fc06df90 2024-09-13T16:43:42.885 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.881+0000 7f2801400640 1 --2- 172.21.15.28:0/711344127 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27fc06dbb0 0x7f27fc1ecc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:42.885 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.881+0000 7f2800a00640 1 --2- 172.21.15.28:0/711344127 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27fc06df90 0x7f27fc14ee00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f27e8031000 tx=0x7f27e80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:42.886 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.881+0000 7f27f9600640 1 -- 172.21.15.28:0/711344127 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e803d830 con 0x7f27fc06df90 2024-09-13T16:43:42.886 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.881+0000 7f27f9600640 1 -- 172.21.15.28:0/711344127 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e803d9b0 con 0x7f27fc06df90 2024-09-13T16:43:42.886 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.881+0000 7f27f9600640 1 -- 172.21.15.28:0/711344127 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e803dd60 con 0x7f27fc06df90 2024-09-13T16:43:42.886 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.881+0000 7f2803e00640 1 -- 172.21.15.28:0/711344127 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e80094d0 con 0x7f27fc06df90 2024-09-13T16:43:42.886 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.881+0000 7f2803e00640 1 -- 172.21.15.28:0/711344127 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27fc0c9370 con 0x7f27fc06df90 2024-09-13T16:43:42.886 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.881+0000 7f27f9600640 1 -- 172.21.15.28:0/711344127 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f27e8039040 con 0x7f27fc06df90 2024-09-13T16:43:42.886 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.882+0000 7f27f9600640 1 -- 172.21.15.28:0/711344127 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f27e8046e20 con 0x7f27fc06df90 2024-09-13T16:43:42.886 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.882+0000 7f2803e00640 1 -- 172.21.15.28:0/711344127 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27fc0c91d0 con 0x7f27fc06df90 2024-09-13T16:43:42.887 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.885+0000 7f27f9600640 1 -- 172.21.15.28:0/711344127 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f27fc0c91d0 con 0x7f27fc06df90 2024-09-13T16:43:42.898 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:42.897+0000 7f98e4c38040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T16:43:42.935 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:42.934+0000 7f26d0638040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T16:43:42.941 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:42.940+0000 7f98e4c38040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T16:43:42.948 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.947+0000 7f2803e00640 1 -- 172.21.15.28:0/711344127 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27fc0c9b70 con 0x7f27fc06df90 2024-09-13T16:43:42.949 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.948+0000 7f27f9600640 1 -- 172.21.15.28:0/711344127 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f27fc0c9b70 con 0x7f27fc06df90 2024-09-13T16:43:42.949 INFO:teuthology.misc.health.smithi028.stdout: 2024-09-13T16:43:42.949 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":10,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:43:41.595256+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:43:42.950 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.949+0000 7f2803e00640 1 -- 172.21.15.28:0/711344127 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27fc06df90 msgr2=0x7f27fc14ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:42.950 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.949+0000 7f2803e00640 1 --2- 172.21.15.28:0/711344127 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27fc06df90 0x7f27fc14ee00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f27e8031000 tx=0x7f27e80046c0 comp rx=0 tx=0).stop 2024-09-13T16:43:42.951 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.949+0000 7f2803e00640 1 -- 172.21.15.28:0/711344127 shutdown_connections 2024-09-13T16:43:42.951 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.949+0000 7f2803e00640 1 --2- 172.21.15.28:0/711344127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27fc06fee0 0x7f27fc0c8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:42.951 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.949+0000 7f2803e00640 1 --2- 172.21.15.28:0/711344127 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f27fc06df90 0x7f27fc14ee00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:42.951 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.949+0000 7f2803e00640 1 --2- 172.21.15.28:0/711344127 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f27fc06dbb0 0x7f27fc1ecc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:42.951 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.949+0000 7f2803e00640 1 -- 172.21.15.28:0/711344127 >> 172.21.15.28:0/711344127 conn(0x7f27fc069e70 msgr2=0x7f27fc0c3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:42.951 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.949+0000 7f2803e00640 1 -- 172.21.15.28:0/711344127 shutdown_connections 2024-09-13T16:43:42.951 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:42.949+0000 7f2803e00640 1 -- 172.21.15.28:0/711344127 wait complete. 2024-09-13T16:43:42.962 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-13T16:43:43.061 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:43.060+0000 7f26d0638040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T16:43:43.118 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:43.116+0000 7f26d0638040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T16:43:43.249 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:43.247+0000 7f26d0638040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T16:43:43.313 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:43.312+0000 7f26d0638040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T16:43:43.368 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:43.366+0000 7f26d0638040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T16:43:43.498 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:43.497+0000 7f26d0638040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T16:43:43.559 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:43.558+0000 7f26d0638040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T16:43:43.578 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:43.577+0000 7f98e4c38040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T16:43:43.622 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:43.621+0000 7f26d0638040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T16:43:43.703 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:43.702+0000 7f26d0638040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T16:43:43.758 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:43.756+0000 7f26d0638040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T16:43:43.812 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:43.810+0000 7f26d0638040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T16:43:44.573 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T16:43:44.571+0000 7f26d0638040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T16:43:44.595 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:44.593+0000 7f9885ed8640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-13T16:43:44.600 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-09-13T16:43:44.599+0000 7f9885ed8640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-13T16:43:45.729 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-09-13T16:43:45.727+0000 7f13425bf640 -1 osd.4 0 log_to_monitors true 2024-09-13T16:43:45.757 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-09-13T16:43:45.755+0000 7eff2eada640 -1 osd.6 0 log_to_monitors true 2024-09-13T16:43:45.773 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-09-13T16:43:45.772+0000 7f0416e9c640 -1 osd.5 0 log_to_monitors true 2024-09-13T16:43:45.783 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-13T16:43:45.783+0000 7f4b60ae6640 -1 osd.7 0 log_to_monitors true 2024-09-13T16:43:45.865 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-13T16:43:45.863+0000 7fec16f23640 -1 osd.0 0 log_to_monitors true 2024-09-13T16:43:45.978 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-13T16:43:45.976+0000 7f32afc20640 -1 osd.1 0 log_to_monitors true 2024-09-13T16:43:45.992 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-13T16:43:45.991+0000 7fc105302640 -1 osd.2 0 log_to_monitors true 2024-09-13T16:43:46.001 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-09-13T16:43:46.000+0000 7fcdf940e640 -1 osd.3 0 log_to_monitors true 2024-09-13T16:43:48.474 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-13T16:43:48.473+0000 7f4b5ca73640 -1 osd.7 0 waiting for initial osdmap 2024-09-13T16:43:48.489 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-09-13T16:43:48.488+0000 7f4b54894640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T16:43:48.496 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-09-13T16:43:48.495+0000 7f0412e29640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T16:43:48.501 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-09-13T16:43:48.500+0000 7f133e54c640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T16:43:48.511 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-09-13T16:43:48.510+0000 7f040ac4a640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T16:43:48.513 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-09-13T16:43:48.512+0000 7f133636d640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T16:43:48.530 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-09-13T16:43:48.529+0000 7eff2aa67640 -1 osd.6 0 waiting for initial osdmap 2024-09-13T16:43:48.546 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-09-13T16:43:48.545+0000 7eff22888640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T16:43:48.992 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-13T16:43:48.991+0000 7fc100200640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T16:43:49.008 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-13T16:43:49.007+0000 7fc0f6200640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T16:43:49.061 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-13T16:43:49.059+0000 7f32aa600640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T16:43:49.071 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-13T16:43:49.070+0000 7f32a0600640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T16:43:49.080 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-09-13T16:43:49.078+0000 7fcdf4600640 -1 osd.3 0 waiting for initial osdmap 2024-09-13T16:43:49.088 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-09-13T16:43:49.087+0000 7fcde9c00640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T16:43:49.108 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-13T16:43:49.106+0000 7fec12800640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T16:43:49.114 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-13T16:43:49.113+0000 7fec07e00640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T16:43:49.768 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T16:43:49.907 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.905+0000 7f883d600640 1 Processor -- start 2024-09-13T16:43:49.907 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.905+0000 7f883d600640 1 -- start start 2024-09-13T16:43:49.908 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.906+0000 7f883d600640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8838150dc0 0x7f88380c22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:49.908 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.906+0000 7f883d600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f88380c10e0 0x7f8838157520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:49.908 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.906+0000 7f883d600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8838159fe0 0x7f88380c5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:49.908 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.906+0000 7f883d600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88380c42e0 con 0x7f88380c10e0 2024-09-13T16:43:49.908 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.906+0000 7f883d600640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8838157090 con 0x7f8838150dc0 2024-09-13T16:43:49.908 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.906+0000 7f883d600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88381511a0 con 0x7f8838159fe0 2024-09-13T16:43:49.909 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.907+0000 7f882fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f88380c10e0 0x7f8838157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:49.909 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.907+0000 7f8837400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8838159fe0 0x7f88380c5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:49.909 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.907+0000 7f8836a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8838150dc0 0x7f88380c22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:49.909 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.907+0000 7f882fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f88380c10e0 0x7f8838157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47248/0 (socket says 172.21.15.28:47248) 2024-09-13T16:43:49.909 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.907+0000 7f882fe00640 1 -- 172.21.15.28:0/2139410823 learned_addr learned my addr 172.21.15.28:0/2139410823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:49.910 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.907+0000 7f882fe00640 1 -- 172.21.15.28:0/2139410823 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8838159fe0 msgr2=0x7f88380c5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:49.910 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.907+0000 7f882fe00640 1 --2- 172.21.15.28:0/2139410823 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8838159fe0 0x7f88380c5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:49.910 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.907+0000 7f882fe00640 1 -- 172.21.15.28:0/2139410823 >> v2:172.21.15.191:3300/0 conn(0x7f8838150dc0 msgr2=0x7f88380c22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:49.910 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.907+0000 7f882fe00640 1 --2- 172.21.15.28:0/2139410823 >> v2:172.21.15.191:3300/0 conn(0x7f8838150dc0 0x7f88380c22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:49.910 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.907+0000 7f882fe00640 1 -- 172.21.15.28:0/2139410823 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f883815a800 con 0x7f88380c10e0 2024-09-13T16:43:49.910 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.908+0000 7f882fe00640 1 --2- 172.21.15.28:0/2139410823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88380c10e0 0x7f8838157520 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f883806e950 tx=0x7f882002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5baaa1fed064470 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:49.910 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.908+0000 7f8836000640 1 -- 172.21.15.28:0/2139410823 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8820032440 con 0x7f88380c10e0 2024-09-13T16:43:49.910 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.909+0000 7f8836000640 1 -- 172.21.15.28:0/2139410823 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88200325a0 con 0x7f88380c10e0 2024-09-13T16:43:49.911 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.909+0000 7f8836000640 1 -- 172.21.15.28:0/2139410823 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8820032930 con 0x7f88380c10e0 2024-09-13T16:43:49.912 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.909+0000 7f883d600640 1 -- 172.21.15.28:0/2139410823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88380c10e0 msgr2=0x7f8838157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:49.912 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.909+0000 7f883d600640 1 --2- 172.21.15.28:0/2139410823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88380c10e0 0x7f8838157520 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f883806e950 tx=0x7f882002a270 comp rx=0 tx=0).stop 2024-09-13T16:43:49.912 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.909+0000 7f883d600640 1 -- 172.21.15.28:0/2139410823 shutdown_connections 2024-09-13T16:43:49.912 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.909+0000 7f883d600640 1 --2- 172.21.15.28:0/2139410823 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8838159fe0 0x7f88380c5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:49.912 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.909+0000 7f883d600640 1 --2- 172.21.15.28:0/2139410823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88380c10e0 0x7f8838157520 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:49.913 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.909+0000 7f883d600640 1 --2- 172.21.15.28:0/2139410823 >> v2:172.21.15.191:3300/0 conn(0x7f8838150dc0 0x7f88380c22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:49.913 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.909+0000 7f883d600640 1 -- 172.21.15.28:0/2139410823 >> 172.21.15.28:0/2139410823 conn(0x7f8838069e70 msgr2=0x7f883806a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:49.913 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.909+0000 7f883d600640 1 -- 172.21.15.28:0/2139410823 shutdown_connections 2024-09-13T16:43:49.913 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.909+0000 7f883d600640 1 -- 172.21.15.28:0/2139410823 wait complete. 2024-09-13T16:43:49.913 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.911+0000 7f883d600640 1 Processor -- start 2024-09-13T16:43:49.913 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.911+0000 7f883d600640 1 -- start start 2024-09-13T16:43:49.915 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.912+0000 7f883d600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88380c10e0 0x7f8838147f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:49.915 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.912+0000 7f883d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8838150dc0 0x7f88381f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:49.915 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.912+0000 7f883d600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8838159fe0 0x7f88380c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:49.915 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.912+0000 7f883d600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88380cb210 con 0x7f8838150dc0 2024-09-13T16:43:49.915 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.912+0000 7f883d600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883814a830 con 0x7f8838159fe0 2024-09-13T16:43:49.916 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.912+0000 7f883d600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f883814c790 con 0x7f88380c10e0 2024-09-13T16:43:49.916 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.912+0000 7f882fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8838150dc0 0x7f88381f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:49.916 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.912+0000 7f8836a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88380c10e0 0x7f8838147f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:49.916 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.912+0000 7f882fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8838150dc0 0x7f88381f0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47260/0 (socket says 172.21.15.28:47260) 2024-09-13T16:43:49.916 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.912+0000 7f8836a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88380c10e0 0x7f8838147f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:57396/0 (socket says 172.21.15.28:57396) 2024-09-13T16:43:49.916 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.912+0000 7f882fe00640 1 -- 172.21.15.28:0/4050962423 learned_addr learned my addr 172.21.15.28:0/4050962423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:49.916 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.912+0000 7f8837400640 1 --2- 172.21.15.28:0/4050962423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8838159fe0 0x7f88380c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:49.916 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.913+0000 7f882fe00640 1 -- 172.21.15.28:0/4050962423 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88380c10e0 msgr2=0x7f8838147f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:49.916 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.913+0000 7f882fe00640 1 --2- 172.21.15.28:0/4050962423 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88380c10e0 0x7f8838147f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:49.916 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.913+0000 7f882fe00640 1 -- 172.21.15.28:0/4050962423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8838159fe0 msgr2=0x7f88380c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:49.916 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.913+0000 7f882fe00640 1 --2- 172.21.15.28:0/4050962423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8838159fe0 0x7f88380c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:49.917 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.913+0000 7f882fe00640 1 -- 172.21.15.28:0/4050962423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88200094d0 con 0x7f8838150dc0 2024-09-13T16:43:49.917 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.913+0000 7f882fe00640 1 --2- 172.21.15.28:0/4050962423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8838150dc0 0x7f88381f0580 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f882002a2b0 tx=0x7f88200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:49.917 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.913+0000 7f882ea00640 1 -- 172.21.15.28:0/4050962423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882003d830 con 0x7f8838150dc0 2024-09-13T16:43:49.917 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.913+0000 7f882ea00640 1 -- 172.21.15.28:0/4050962423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882003d990 con 0x7f8838150dc0 2024-09-13T16:43:49.917 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.913+0000 7f883d600640 1 -- 172.21.15.28:0/4050962423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f883814ca70 con 0x7f8838150dc0 2024-09-13T16:43:49.917 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.913+0000 7f882ea00640 1 -- 172.21.15.28:0/4050962423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882003dd40 con 0x7f8838150dc0 2024-09-13T16:43:49.917 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.914+0000 7f883d600640 1 -- 172.21.15.28:0/4050962423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88381efaa0 con 0x7f8838150dc0 2024-09-13T16:43:49.917 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.915+0000 7f883d600640 1 -- 172.21.15.28:0/4050962423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8838066c80 con 0x7f8838150dc0 2024-09-13T16:43:49.918 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.916+0000 7f882ea00640 1 -- 172.21.15.28:0/4050962423 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7f8820039040 con 0x7f8838150dc0 2024-09-13T16:43:49.919 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.917+0000 7f882ea00640 1 --2- 172.21.15.28:0/4050962423 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f881406ab70 0x7f881406d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:49.919 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.917+0000 7f882ea00640 1 -- 172.21.15.28:0/4050962423 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f88200a9c40 con 0x7f8838150dc0 2024-09-13T16:43:49.919 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.917+0000 7f8836a00640 1 --2- 172.21.15.28:0/4050962423 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f881406ab70 0x7f881406d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:49.919 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.917+0000 7f8836a00640 1 --2- 172.21.15.28:0/4050962423 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f881406ab70 0x7f881406d030 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8828009ac0 tx=0x7f8828009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:49.922 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:49.921+0000 7f882ea00640 1 -- 172.21.15.28:0/4050962423 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8820075460 con 0x7f8838150dc0 2024-09-13T16:43:50.102 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:50.100+0000 7f883d600640 1 -- 172.21.15.28:0/4050962423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f883815a490 con 0x7f8838150dc0 2024-09-13T16:43:50.103 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:50.101+0000 7f882ea00640 1 -- 172.21.15.28:0/4050962423 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10884 (secure 0 0 0) 0x7f8820078a80 con 0x7f8838150dc0 2024-09-13T16:43:50.103 INFO:teuthology.misc.health.smithi028.stdout: 2024-09-13T16:43:50.104 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":13,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:43:49.476313+0000","last_up_change":"2024-09-13T16:43:49.476313+0000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6801","nonce":2964323435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6803","nonce":2964323435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6807","nonce":2964323435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6805","nonce":2964323435}]},"public_addr":"172.21.15.28:6801/2964323435","cluster_addr":"172.21.15.28:6803/2964323435","heartbeat_back_addr":"172.21.15.28:6807/2964323435","heartbeat_front_addr":"172.21.15.28:6805/2964323435","state":["exists","up"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6809","nonce":23768570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6811","nonce":23768570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6815","nonce":23768570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6813","nonce":23768570}]},"public_addr":"172.21.15.28:6809/23768570","cluster_addr":"172.21.15.28:6811/23768570","heartbeat_back_addr":"172.21.15.28:6815/23768570","heartbeat_front_addr":"172.21.15.28:6813/23768570","state":["exists","up"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6817","nonce":886887303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6819","nonce":886887303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6823","nonce":886887303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6821","nonce":886887303}]},"public_addr":"172.21.15.28:6817/886887303","cluster_addr":"172.21.15.28:6819/886887303","heartbeat_back_addr":"172.21.15.28:6823/886887303","heartbeat_front_addr":"172.21.15.28:6821/886887303","state":["exists","up"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6825","nonce":2616036674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6827","nonce":2616036674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6831","nonce":2616036674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6829","nonce":2616036674}]},"public_addr":"172.21.15.28:6825/2616036674","cluster_addr":"172.21.15.28:6827/2616036674","heartbeat_back_addr":"172.21.15.28:6831/2616036674","heartbeat_front_addr":"172.21.15.28:6829/2616036674","state":["exists","up"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6801","nonce":2817102450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6803","nonce":2817102450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6807","nonce":2817102450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6805","nonce":2817102450}]},"public_addr":"172.21.15.191:6801/2817102450","cluster_addr":"172.21.15.191:6803/2817102450","heartbeat_back_addr":"172.21.15.191:6807/2817102450","heartbeat_front_addr":"172.21.15.191:6805/2817102450","state":["exists","up"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6817","nonce":2503027971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6819","nonce":2503027971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6823","nonce":2503027971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6821","nonce":2503027971}]},"public_addr":"172.21.15.191:6817/2503027971","cluster_addr":"172.21.15.191:6819/2503027971","heartbeat_back_addr":"172.21.15.191:6823/2503027971","heartbeat_front_addr":"172.21.15.191:6821/2503027971","state":["exists","up"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6809","nonce":3006595088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6811","nonce":3006595088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6815","nonce":3006595088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6813","nonce":3006595088}]},"public_addr":"172.21.15.191:6809/3006595088","cluster_addr":"172.21.15.191:6811/3006595088","heartbeat_back_addr":"172.21.15.191:6815/3006595088","heartbeat_front_addr":"172.21.15.191:6813/3006595088","state":["exists","up"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6825","nonce":1477327720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6827","nonce":1477327720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6831","nonce":1477327720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6829","nonce":1477327720}]},"public_addr":"172.21.15.191:6825/1477327720","cluster_addr":"172.21.15.191:6827/1477327720","heartbeat_back_addr":"172.21.15.191:6831/1477327720","heartbeat_front_addr":"172.21.15.191:6829/1477327720","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:43:50.105 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:50.104+0000 7f883d600640 1 -- 172.21.15.28:0/4050962423 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f881406ab70 msgr2=0x7f881406d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:50.106 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:50.104+0000 7f883d600640 1 --2- 172.21.15.28:0/4050962423 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f881406ab70 0x7f881406d030 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8828009ac0 tx=0x7f8828009190 comp rx=0 tx=0).stop 2024-09-13T16:43:50.106 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:50.104+0000 7f883d600640 1 -- 172.21.15.28:0/4050962423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8838150dc0 msgr2=0x7f88381f0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:50.106 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:50.104+0000 7f883d600640 1 --2- 172.21.15.28:0/4050962423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8838150dc0 0x7f88381f0580 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f882002a2b0 tx=0x7f88200046c0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.106 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:50.104+0000 7f883d600640 1 -- 172.21.15.28:0/4050962423 shutdown_connections 2024-09-13T16:43:50.106 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:50.104+0000 7f883d600640 1 --2- 172.21.15.28:0/4050962423 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f881406ab70 0x7f881406d030 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.106 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:50.104+0000 7f883d600640 1 --2- 172.21.15.28:0/4050962423 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8838159fe0 0x7f88380c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.106 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:50.104+0000 7f883d600640 1 --2- 172.21.15.28:0/4050962423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8838150dc0 0x7f88381f0580 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.106 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:50.104+0000 7f883d600640 1 --2- 172.21.15.28:0/4050962423 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88380c10e0 0x7f8838147f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.106 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:50.104+0000 7f883d600640 1 -- 172.21.15.28:0/4050962423 >> 172.21.15.28:0/4050962423 conn(0x7f8838069e70 msgr2=0x7f88380c18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:50.106 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:50.104+0000 7f883d600640 1 -- 172.21.15.28:0/4050962423 shutdown_connections 2024-09-13T16:43:50.106 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T16:43:50.104+0000 7f883d600640 1 -- 172.21.15.28:0/4050962423 wait complete. 2024-09-13T16:43:50.118 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-13T16:43:50.118 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-13T16:43:50.118 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T16:43:50.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T16:43:50.123 INFO:tasks.ceph.mds.a:Started 2024-09-13T16:43:50.123 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T16:43:50.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T16:43:50.127 INFO:tasks.ceph.mds.c:Started 2024-09-13T16:43:50.127 INFO:tasks.ceph.mds.e:Restarting daemon 2024-09-13T16:43:50.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-09-13T16:43:50.130 INFO:tasks.ceph.mds.e:Started 2024-09-13T16:43:50.130 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T16:43:50.131 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T16:43:50.133 INFO:tasks.ceph.mds.b:Started 2024-09-13T16:43:50.133 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T16:43:50.134 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T16:43:50.136 INFO:tasks.ceph.mds.d:Started 2024-09-13T16:43:50.136 INFO:tasks.ceph.mds.f:Restarting daemon 2024-09-13T16:43:50.136 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-09-13T16:43:50.138 INFO:tasks.ceph.mds.f:Started 2024-09-13T16:43:50.138 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-09-13T16:43:50.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:43:50.219 INFO:tasks.ceph.mds.d.smithi191.stdout:starting mds.d at 2024-09-13T16:43:50.219 INFO:tasks.ceph.mds.b.smithi191.stdout:starting mds.b at 2024-09-13T16:43:50.220 INFO:tasks.ceph.mds.f.smithi191.stdout:starting mds.f at 2024-09-13T16:43:50.266 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T16:43:50.267 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T16:43:50.277 INFO:tasks.ceph.mds.e.smithi028.stdout:starting mds.e at 2024-09-13T16:43:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.308+0000 7f411ce00640 1 Processor -- start 2024-09-13T16:43:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.308+0000 7f411ce00640 1 -- start start 2024-09-13T16:43:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.309+0000 7f411ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f41180c46f0 0x7f41180ca940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.309+0000 7f411ce00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4118159500 0x7f41180ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.309+0000 7f411ce00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41180cbbe0 con 0x7f41180ce7f0 2024-09-13T16:43:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.309+0000 7f411ce00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41180cae80 con 0x7f41180c46f0 2024-09-13T16:43:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.309+0000 7f411ce00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41181532e0 con 0x7f4118159500 2024-09-13T16:43:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.309+0000 7f4117400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f41180c46f0 0x7f41180ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.309+0000 7f4117e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f41180ce7f0 0x7f41180c9940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34322/0 (socket says 172.21.15.28:34322) 2024-09-13T16:43:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.309+0000 7f4117e00640 1 -- 172.21.15.28:0/3176682902 learned_addr learned my addr 172.21.15.28:0/3176682902 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:43:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.309+0000 7f410fe00640 1 --2- 172.21.15.28:0/3176682902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4118159500 0x7f41180ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.310+0000 7f410fe00640 1 -- 172.21.15.28:0/3176682902 >> v2:172.21.15.28:3300/0 conn(0x7f41180c46f0 msgr2=0x7f41180ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.310+0000 7f410fe00640 1 --2- 172.21.15.28:0/3176682902 >> v2:172.21.15.28:3300/0 conn(0x7f41180c46f0 0x7f41180ca940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.310+0000 7f410fe00640 1 -- 172.21.15.28:0/3176682902 >> v1:172.21.15.28:6789/0 conn(0x7f41180ce7f0 legacy=0x7f41180c9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.310+0000 7f410fe00640 1 -- 172.21.15.28:0/3176682902 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41180cb060 con 0x7f4118159500 2024-09-13T16:43:50.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.310+0000 7f4117400640 1 --2- 172.21.15.28:0/3176682902 >> v2:172.21.15.28:3300/0 conn(0x7f41180c46f0 0x7f41180ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:50.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.311+0000 7f410fe00640 1 --2- 172.21.15.28:0/3176682902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4118159500 0x7f41180ce2b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f41000097a0 tx=0x7f410002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57eb17cdb724ac54 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:50.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.311+0000 7f4116a00640 1 -- 172.21.15.28:0/3176682902 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4100032440 con 0x7f4118159500 2024-09-13T16:43:50.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.311+0000 7f4116a00640 1 -- 172.21.15.28:0/3176682902 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41000325a0 con 0x7f4118159500 2024-09-13T16:43:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.311+0000 7f4116a00640 1 -- 172.21.15.28:0/3176682902 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4100032930 con 0x7f4118159500 2024-09-13T16:43:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.312+0000 7f411ce00640 1 -- 172.21.15.28:0/3176682902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4118159500 msgr2=0x7f41180ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.312+0000 7f411ce00640 1 --2- 172.21.15.28:0/3176682902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4118159500 0x7f41180ce2b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f41000097a0 tx=0x7f410002a180 comp rx=0 tx=0).stop 2024-09-13T16:43:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.312+0000 7f411ce00640 1 -- 172.21.15.28:0/3176682902 shutdown_connections 2024-09-13T16:43:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.312+0000 7f411ce00640 1 --2- 172.21.15.28:0/3176682902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4118159500 0x7f41180ce2b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.312+0000 7f411ce00640 1 --2- 172.21.15.28:0/3176682902 >> v2:172.21.15.28:3300/0 conn(0x7f41180c46f0 0x7f41180ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.312+0000 7f411ce00640 1 -- 172.21.15.28:0/3176682902 >> 172.21.15.28:0/3176682902 conn(0x7f411806c830 msgr2=0x7f411806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.312+0000 7f411ce00640 1 -- 172.21.15.28:0/3176682902 shutdown_connections 2024-09-13T16:43:50.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.312+0000 7f411ce00640 1 -- 172.21.15.28:0/3176682902 wait complete. 2024-09-13T16:43:50.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.313+0000 7f411ce00640 1 Processor -- start 2024-09-13T16:43:50.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.313+0000 7f411ce00640 1 -- start start 2024-09-13T16:43:50.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.314+0000 7f411ce00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41180c46f0 0x7f411814ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:50.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.314+0000 7f411ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41180ce7f0 0x7f4118148450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:50.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.314+0000 7f411ce00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4118159500 0x7f4118156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:50.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.314+0000 7f411ce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411814b740 con 0x7f41180ce7f0 2024-09-13T16:43:50.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.314+0000 7f411ce00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41181eed60 con 0x7f4118159500 2024-09-13T16:43:50.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.314+0000 7f411ce00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411814b4d0 con 0x7f41180c46f0 2024-09-13T16:43:50.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.314+0000 7f410fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41180ce7f0 0x7f4118148450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:50.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.315+0000 7f410fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41180ce7f0 0x7f4118148450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47326/0 (socket says 172.21.15.28:47326) 2024-09-13T16:43:50.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.315+0000 7f410fe00640 1 -- 172.21.15.28:0/3457232421 learned_addr learned my addr 172.21.15.28:0/3457232421 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:50.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.315+0000 7f4117e00640 1 --2- 172.21.15.28:0/3457232421 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4118159500 0x7f4118156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:50.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.315+0000 7f4117400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41180c46f0 0x7f411814ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:50.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.315+0000 7f410fe00640 1 -- 172.21.15.28:0/3457232421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41180c46f0 msgr2=0x7f411814ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:50.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.315+0000 7f410fe00640 1 --2- 172.21.15.28:0/3457232421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41180c46f0 0x7f411814ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.315+0000 7f410fe00640 1 -- 172.21.15.28:0/3457232421 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4118159500 msgr2=0x7f4118156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:50.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.315+0000 7f410fe00640 1 --2- 172.21.15.28:0/3457232421 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4118159500 0x7f4118156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.315+0000 7f410fe00640 1 -- 172.21.15.28:0/3457232421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4108009400 con 0x7f41180ce7f0 2024-09-13T16:43:50.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.315+0000 7f4117400640 1 --2- 172.21.15.28:0/3457232421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41180c46f0 0x7f411814ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:50.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.315+0000 7f410fe00640 1 --2- 172.21.15.28:0/3457232421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41180ce7f0 0x7f4118148450 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f410002a770 tx=0x7f41000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:50.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.316+0000 7f410f400640 1 -- 172.21.15.28:0/3457232421 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4100006e10 con 0x7f41180ce7f0 2024-09-13T16:43:50.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.316+0000 7f410f400640 1 -- 172.21.15.28:0/3457232421 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4100032e90 con 0x7f41180ce7f0 2024-09-13T16:43:50.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.316+0000 7f410f400640 1 -- 172.21.15.28:0/3457232421 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4100030ba0 con 0x7f41180ce7f0 2024-09-13T16:43:50.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.316+0000 7f411ce00640 1 -- 172.21.15.28:0/3457232421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4100009400 con 0x7f41180ce7f0 2024-09-13T16:43:50.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.316+0000 7f411ce00640 1 -- 172.21.15.28:0/3457232421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41180c65e0 con 0x7f41180ce7f0 2024-09-13T16:43:50.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.317+0000 7f411ce00640 1 -- 172.21.15.28:0/3457232421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4118067460 con 0x7f41180ce7f0 2024-09-13T16:43:50.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.318+0000 7f410f400640 1 -- 172.21.15.28:0/3457232421 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7f4100039040 con 0x7f41180ce7f0 2024-09-13T16:43:50.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.319+0000 7f410f400640 1 --2- 172.21.15.28:0/3457232421 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f40f406ab70 0x7f40f406d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:50.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.319+0000 7f410f400640 1 -- 172.21.15.28:0/3457232421 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f41000a9500 con 0x7f41180ce7f0 2024-09-13T16:43:50.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.319+0000 7f4117400640 1 --2- 172.21.15.28:0/3457232421 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f40f406ab70 0x7f40f406d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:50.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.320+0000 7f4117400640 1 --2- 172.21.15.28:0/3457232421 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f40f406ab70 0x7f40f406d030 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4108011380 tx=0x7f4108009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.323+0000 7f410f400640 1 -- 172.21.15.28:0/3457232421 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4100074e50 con 0x7f41180ce7f0 2024-09-13T16:43:50.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.507+0000 7f411ce00640 1 -- 172.21.15.28:0/3457232421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41180c3d60 con 0x7f41180ce7f0 2024-09-13T16:43:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.508+0000 7f410f400640 1 -- 172.21.15.28:0/3457232421 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+12830 (secure 0 0 0) 0x7f41180c3d60 con 0x7f41180ce7f0 2024-09-13T16:43:50.510 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:50.510 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":14,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:43:50.498116+0000","last_up_change":"2024-09-13T16:43:49.476313+0000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:43:49.643606+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6801","nonce":2964323435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6803","nonce":2964323435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6807","nonce":2964323435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6805","nonce":2964323435}]},"public_addr":"172.21.15.28:6801/2964323435","cluster_addr":"172.21.15.28:6803/2964323435","heartbeat_back_addr":"172.21.15.28:6807/2964323435","heartbeat_front_addr":"172.21.15.28:6805/2964323435","state":["exists","up"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6809","nonce":23768570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6811","nonce":23768570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6815","nonce":23768570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6813","nonce":23768570}]},"public_addr":"172.21.15.28:6809/23768570","cluster_addr":"172.21.15.28:6811/23768570","heartbeat_back_addr":"172.21.15.28:6815/23768570","heartbeat_front_addr":"172.21.15.28:6813/23768570","state":["exists","up"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6817","nonce":886887303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6819","nonce":886887303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6823","nonce":886887303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6821","nonce":886887303}]},"public_addr":"172.21.15.28:6817/886887303","cluster_addr":"172.21.15.28:6819/886887303","heartbeat_back_addr":"172.21.15.28:6823/886887303","heartbeat_front_addr":"172.21.15.28:6821/886887303","state":["exists","up"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6825","nonce":2616036674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6827","nonce":2616036674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6831","nonce":2616036674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6829","nonce":2616036674}]},"public_addr":"172.21.15.28:6825/2616036674","cluster_addr":"172.21.15.28:6827/2616036674","heartbeat_back_addr":"172.21.15.28:6831/2616036674","heartbeat_front_addr":"172.21.15.28:6829/2616036674","state":["exists","up"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6801","nonce":2817102450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6803","nonce":2817102450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6807","nonce":2817102450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6805","nonce":2817102450}]},"public_addr":"172.21.15.191:6801/2817102450","cluster_addr":"172.21.15.191:6803/2817102450","heartbeat_back_addr":"172.21.15.191:6807/2817102450","heartbeat_front_addr":"172.21.15.191:6805/2817102450","state":["exists","up"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6817","nonce":2503027971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6819","nonce":2503027971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6823","nonce":2503027971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6821","nonce":2503027971}]},"public_addr":"172.21.15.191:6817/2503027971","cluster_addr":"172.21.15.191:6819/2503027971","heartbeat_back_addr":"172.21.15.191:6823/2503027971","heartbeat_front_addr":"172.21.15.191:6821/2503027971","state":["exists","up"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6809","nonce":3006595088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6811","nonce":3006595088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6815","nonce":3006595088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6813","nonce":3006595088}]},"public_addr":"172.21.15.191:6809/3006595088","cluster_addr":"172.21.15.191:6811/3006595088","heartbeat_back_addr":"172.21.15.191:6815/3006595088","heartbeat_front_addr":"172.21.15.191:6813/3006595088","state":["exists","up"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6825","nonce":1477327720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6827","nonce":1477327720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6831","nonce":1477327720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6829","nonce":1477327720}]},"public_addr":"172.21.15.191:6825/1477327720","cluster_addr":"172.21.15.191:6827/1477327720","heartbeat_back_addr":"172.21.15.191:6831/1477327720","heartbeat_front_addr":"172.21.15.191:6829/1477327720","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.917250+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.954351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.017329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.046221+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.707467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.726947+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.782321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.764850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:43:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.510+0000 7f411ce00640 1 -- 172.21.15.28:0/3457232421 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f40f406ab70 msgr2=0x7f40f406d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.510+0000 7f411ce00640 1 --2- 172.21.15.28:0/3457232421 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f40f406ab70 0x7f40f406d030 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4108011380 tx=0x7f4108009190 comp rx=0 tx=0).stop 2024-09-13T16:43:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.510+0000 7f411ce00640 1 -- 172.21.15.28:0/3457232421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41180ce7f0 msgr2=0x7f4118148450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.510+0000 7f411ce00640 1 --2- 172.21.15.28:0/3457232421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41180ce7f0 0x7f4118148450 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f410002a770 tx=0x7f41000046c0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.510+0000 7f411ce00640 1 -- 172.21.15.28:0/3457232421 shutdown_connections 2024-09-13T16:43:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.510+0000 7f411ce00640 1 --2- 172.21.15.28:0/3457232421 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f40f406ab70 0x7f40f406d030 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.510+0000 7f411ce00640 1 --2- 172.21.15.28:0/3457232421 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4118159500 0x7f4118156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.510+0000 7f411ce00640 1 --2- 172.21.15.28:0/3457232421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41180ce7f0 0x7f4118148450 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.510+0000 7f411ce00640 1 --2- 172.21.15.28:0/3457232421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41180c46f0 0x7f411814ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.510+0000 7f411ce00640 1 -- 172.21.15.28:0/3457232421 >> 172.21.15.28:0/3457232421 conn(0x7f411806c830 msgr2=0x7f41180c9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.510+0000 7f411ce00640 1 -- 172.21.15.28:0/3457232421 shutdown_connections 2024-09-13T16:43:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.510+0000 7f411ce00640 1 -- 172.21.15.28:0/3457232421 wait complete. 2024-09-13T16:43:50.527 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:43:49.643606+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:43:50.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:43:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.684+0000 7fbb86600640 1 Processor -- start 2024-09-13T16:43:50.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.684+0000 7fbb86600640 1 -- start start 2024-09-13T16:43:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.685+0000 7fbb86600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb800afd50 0x7fbb800c8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.685+0000 7fbb86600640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbb800b0130 0x7fbb8006cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.685+0000 7fbb86600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb800ca170 con 0x7fbb800ca8e0 2024-09-13T16:43:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.685+0000 7fbb86600640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb800c91c0 con 0x7fbb800b0130 2024-09-13T16:43:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.685+0000 7fbb86600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8006d2c0 con 0x7fbb800afd50 2024-09-13T16:43:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.685+0000 7fbb7fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbb800ca8e0 0x7fbb80152b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34336/0 (socket says 172.21.15.28:34336) 2024-09-13T16:43:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.685+0000 7fbb7fe00640 1 -- 172.21.15.28:0/3176972268 learned_addr learned my addr 172.21.15.28:0/3176972268 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:43:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.685+0000 7fbb77e00640 1 --2- 172.21.15.28:0/3176972268 >> v2:172.21.15.191:3300/0 conn(0x7fbb800b0130 0x7fbb8006cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.685+0000 7fbb7f400640 1 --2- 172.21.15.28:0/3176972268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb800afd50 0x7fbb800c8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:50.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.686+0000 7fbb7ea00640 1 -- 172.21.15.28:0/3176972268 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227831507 0 0) 0x7fbb800ca170 con 0x7fbb800ca8e0 2024-09-13T16:43:50.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.686+0000 7fbb7ea00640 1 -- 172.21.15.28:0/3176972268 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb64003520 con 0x7fbb800ca8e0 2024-09-13T16:43:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.686+0000 7fbb77e00640 1 -- 172.21.15.28:0/3176972268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb800afd50 msgr2=0x7fbb800c8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.686+0000 7fbb77e00640 1 --2- 172.21.15.28:0/3176972268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb800afd50 0x7fbb800c8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.686+0000 7fbb77e00640 1 -- 172.21.15.28:0/3176972268 >> v1:172.21.15.28:6789/0 conn(0x7fbb800ca8e0 legacy=0x7fbb80152b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.686+0000 7fbb77e00640 1 -- 172.21.15.28:0/3176972268 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8006c3e0 con 0x7fbb800b0130 2024-09-13T16:43:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.687+0000 7fbb77e00640 1 --2- 172.21.15.28:0/3176972268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb800b0130 0x7fbb8006cd80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbb680097a0 tx=0x7fbb6802a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2207dbafdbdbc8c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.687+0000 7fbb7ea00640 1 -- 172.21.15.28:0/3176972268 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6802ae50 con 0x7fbb800b0130 2024-09-13T16:43:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.687+0000 7fbb7ea00640 1 -- 172.21.15.28:0/3176972268 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb68032440 con 0x7fbb800b0130 2024-09-13T16:43:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.687+0000 7fbb7ea00640 1 -- 172.21.15.28:0/3176972268 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb680327b0 con 0x7fbb800b0130 2024-09-13T16:43:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.687+0000 7fbb86600640 1 -- 172.21.15.28:0/3176972268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb800b0130 msgr2=0x7fbb8006cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.687+0000 7fbb86600640 1 --2- 172.21.15.28:0/3176972268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb800b0130 0x7fbb8006cd80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbb680097a0 tx=0x7fbb6802a100 comp rx=0 tx=0).stop 2024-09-13T16:43:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.688+0000 7fbb86600640 1 -- 172.21.15.28:0/3176972268 shutdown_connections 2024-09-13T16:43:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.688+0000 7fbb86600640 1 --2- 172.21.15.28:0/3176972268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb800b0130 0x7fbb8006cd80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.688+0000 7fbb86600640 1 --2- 172.21.15.28:0/3176972268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb800afd50 0x7fbb800c8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.688+0000 7fbb86600640 1 -- 172.21.15.28:0/3176972268 >> 172.21.15.28:0/3176972268 conn(0x7fbb800cad80 msgr2=0x7fbb800cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:50.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.688+0000 7fbb86600640 1 -- 172.21.15.28:0/3176972268 shutdown_connections 2024-09-13T16:43:50.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.688+0000 7fbb86600640 1 -- 172.21.15.28:0/3176972268 wait complete. 2024-09-13T16:43:50.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.689+0000 7fbb86600640 1 Processor -- start 2024-09-13T16:43:50.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.689+0000 7fbb86600640 1 -- start start 2024-09-13T16:43:50.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.690+0000 7fbb86600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb800afd50 0x7fbb801556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:50.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.690+0000 7fbb86600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb800b0130 0x7fbb80155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:50.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.690+0000 7fbb86600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb800ca8e0 0x7fbb80153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:50.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.690+0000 7fbb86600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb800af940 con 0x7fbb800ca8e0 2024-09-13T16:43:50.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.690+0000 7fbb86600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb800cb1f0 con 0x7fbb800afd50 2024-09-13T16:43:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.690+0000 7fbb86600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb800af6a0 con 0x7fbb800b0130 2024-09-13T16:43:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.690+0000 7fbb7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb800ca8e0 0x7fbb80153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.691+0000 7fbb7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb800ca8e0 0x7fbb80153f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47332/0 (socket says 172.21.15.28:47332) 2024-09-13T16:43:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.691+0000 7fbb7fe00640 1 -- 172.21.15.28:0/1884606311 learned_addr learned my addr 172.21.15.28:0/1884606311 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.691+0000 7fbb7f400640 1 --2- 172.21.15.28:0/1884606311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb800afd50 0x7fbb801556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.691+0000 7fbb77e00640 1 --2- 172.21.15.28:0/1884606311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb800b0130 0x7fbb80155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.691+0000 7fbb7fe00640 1 -- 172.21.15.28:0/1884606311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb800b0130 msgr2=0x7fbb80155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.691+0000 7fbb7fe00640 1 --2- 172.21.15.28:0/1884606311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb800b0130 0x7fbb80155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.691+0000 7fbb7fe00640 1 -- 172.21.15.28:0/1884606311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb800afd50 msgr2=0x7fbb801556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.691+0000 7fbb7fe00640 1 --2- 172.21.15.28:0/1884606311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb800afd50 0x7fbb801556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.691+0000 7fbb7fe00640 1 -- 172.21.15.28:0/1884606311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb68009400 con 0x7fbb800ca8e0 2024-09-13T16:43:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.691+0000 7fbb7fe00640 1 --2- 172.21.15.28:0/1884606311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb800ca8e0 0x7fbb80153f50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbb7800c9c0 tx=0x7fbb7800ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:50.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.691+0000 7fbb77400640 1 -- 172.21.15.28:0/1884606311 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7800ab70 con 0x7fbb800ca8e0 2024-09-13T16:43:50.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.692+0000 7fbb77400640 1 -- 172.21.15.28:0/1884606311 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb7800acd0 con 0x7fbb800ca8e0 2024-09-13T16:43:50.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.692+0000 7fbb86600640 1 -- 172.21.15.28:0/1884606311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb801eefc0 con 0x7fbb800ca8e0 2024-09-13T16:43:50.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.692+0000 7fbb77400640 1 -- 172.21.15.28:0/1884606311 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb78012600 con 0x7fbb800ca8e0 2024-09-13T16:43:50.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.692+0000 7fbb86600640 1 -- 172.21.15.28:0/1884606311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb800c68c0 con 0x7fbb800ca8e0 2024-09-13T16:43:50.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.693+0000 7fbb86600640 1 -- 172.21.15.28:0/1884606311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb8006cf60 con 0x7fbb800ca8e0 2024-09-13T16:43:50.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.694+0000 7fbb77400640 1 -- 172.21.15.28:0/1884606311 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7fbb78012760 con 0x7fbb800ca8e0 2024-09-13T16:43:50.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.694+0000 7fbb77400640 1 --2- 172.21.15.28:0/1884606311 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fbb5c06ab70 0x7fbb5c06d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:50.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.695+0000 7fbb77400640 1 -- 172.21.15.28:0/1884606311 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4846+0+0 (secure 0 0 0) 0x7fbb78088ca0 con 0x7fbb800ca8e0 2024-09-13T16:43:50.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.695+0000 7fbb7f400640 1 --2- 172.21.15.28:0/1884606311 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fbb5c06ab70 0x7fbb5c06d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:50.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.695+0000 7fbb7f400640 1 --2- 172.21.15.28:0/1884606311 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fbb5c06ab70 0x7fbb5c06d030 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009b40 tx=0x7fbb70009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:50.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.699+0000 7fbb77400640 1 -- 172.21.15.28:0/1884606311 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbb78054370 con 0x7fbb800ca8e0 2024-09-13T16:43:50.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.876+0000 7fbb86600640 1 -- 172.21.15.28:0/1884606311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbb800c8d20 con 0x7fbb800ca8e0 2024-09-13T16:43:50.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.876+0000 7fbb77400640 1 -- 172.21.15.28:0/1884606311 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v14) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbb78057990 con 0x7fbb800ca8e0 2024-09-13T16:43:50.878 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T16:43:50.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.878+0000 7fbb86600640 1 -- 172.21.15.28:0/1884606311 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fbb5c06ab70 msgr2=0x7fbb5c06d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:50.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.878+0000 7fbb86600640 1 --2- 172.21.15.28:0/1884606311 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fbb5c06ab70 0x7fbb5c06d030 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009b40 tx=0x7fbb70009190 comp rx=0 tx=0).stop 2024-09-13T16:43:50.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.878+0000 7fbb86600640 1 -- 172.21.15.28:0/1884606311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb800ca8e0 msgr2=0x7fbb80153f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:50.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.878+0000 7fbb86600640 1 --2- 172.21.15.28:0/1884606311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb800ca8e0 0x7fbb80153f50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbb7800c9c0 tx=0x7fbb7800ce80 comp rx=0 tx=0).stop 2024-09-13T16:43:50.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.879+0000 7fbb86600640 1 -- 172.21.15.28:0/1884606311 shutdown_connections 2024-09-13T16:43:50.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.879+0000 7fbb86600640 1 --2- 172.21.15.28:0/1884606311 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fbb5c06ab70 0x7fbb5c06d030 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.879+0000 7fbb86600640 1 --2- 172.21.15.28:0/1884606311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb800ca8e0 0x7fbb80153f50 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.879+0000 7fbb86600640 1 --2- 172.21.15.28:0/1884606311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb800b0130 0x7fbb80155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.879+0000 7fbb86600640 1 --2- 172.21.15.28:0/1884606311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb800afd50 0x7fbb801556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:50.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.879+0000 7fbb86600640 1 -- 172.21.15.28:0/1884606311 >> 172.21.15.28:0/1884606311 conn(0x7fbb800cad80 msgr2=0x7fbb800706c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:50.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.879+0000 7fbb86600640 1 -- 172.21.15.28:0/1884606311 shutdown_connections 2024-09-13T16:43:50.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:50.879+0000 7fbb86600640 1 -- 172.21.15.28:0/1884606311 wait complete. 2024-09-13T16:43:50.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:43:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.048+0000 7f3543e00640 1 Processor -- start 2024-09-13T16:43:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.048+0000 7f3543e00640 1 -- start start 2024-09-13T16:43:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.049+0000 7f3543e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f353c0c3670 0x7f353c0c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.049+0000 7f3543e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353c14ca00 con 0x7f353c0c1ea0 2024-09-13T16:43:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.049+0000 7f3543e00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353c155990 con 0x7f353c0c5bd0 2024-09-13T16:43:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.049+0000 7f3543e00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f353c0c2dc0 con 0x7f353c0c3670 2024-09-13T16:43:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.049+0000 7f3541e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f353c0c1ea0 0x7f353c0c6ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34340/0 (socket says 172.21.15.28:34340) 2024-09-13T16:43:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.049+0000 7f3541e00640 1 -- 172.21.15.28:0/2462185774 learned_addr learned my addr 172.21.15.28:0/2462185774 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:43:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.049+0000 7f3540a00640 1 --2- 172.21.15.28:0/2462185774 >> v2:172.21.15.191:3300/0 conn(0x7f353c0c3670 0x7f353c0c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:51.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.050+0000 7f3540a00640 1 -- 172.21.15.28:0/2462185774 >> v1:172.21.15.191:6789/0 conn(0x7f353c0c5bd0 legacy=0x7f353c156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:51.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.050+0000 7f353be00640 1 -- 172.21.15.28:0/2462185774 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679831688 0 0) 0x7f353c14ca00 con 0x7f353c0c1ea0 2024-09-13T16:43:51.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.050+0000 7f3540a00640 1 -- 172.21.15.28:0/2462185774 >> v1:172.21.15.28:6789/0 conn(0x7f353c0c1ea0 legacy=0x7f353c0c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:51.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.050+0000 7f3540a00640 1 -- 172.21.15.28:0/2462185774 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353c0c3b60 con 0x7f353c0c3670 2024-09-13T16:43:51.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.051+0000 7f3540a00640 1 --2- 172.21.15.28:0/2462185774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f353c0c3670 0x7f353c0c1960 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3528009720 tx=0x7f352802a150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96a9c665ed9bf192 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:51.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.051+0000 7f353be00640 1 -- 172.21.15.28:0/2462185774 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352802ae60 con 0x7f353c0c3670 2024-09-13T16:43:51.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.051+0000 7f353be00640 1 -- 172.21.15.28:0/2462185774 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3528032440 con 0x7f353c0c3670 2024-09-13T16:43:51.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.051+0000 7f353be00640 1 -- 172.21.15.28:0/2462185774 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35280327d0 con 0x7f353c0c3670 2024-09-13T16:43:51.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.052+0000 7f3543e00640 1 -- 172.21.15.28:0/2462185774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f353c0c3670 msgr2=0x7f353c0c1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:51.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.052+0000 7f3543e00640 1 --2- 172.21.15.28:0/2462185774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f353c0c3670 0x7f353c0c1960 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3528009720 tx=0x7f352802a150 comp rx=0 tx=0).stop 2024-09-13T16:43:51.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.052+0000 7f3543e00640 1 -- 172.21.15.28:0/2462185774 shutdown_connections 2024-09-13T16:43:51.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.052+0000 7f3543e00640 1 --2- 172.21.15.28:0/2462185774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f353c0c3670 0x7f353c0c1960 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.052+0000 7f3543e00640 1 -- 172.21.15.28:0/2462185774 >> 172.21.15.28:0/2462185774 conn(0x7f353c066040 msgr2=0x7f353c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:51.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.052+0000 7f3543e00640 1 -- 172.21.15.28:0/2462185774 shutdown_connections 2024-09-13T16:43:51.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.052+0000 7f3543e00640 1 -- 172.21.15.28:0/2462185774 wait complete. 2024-09-13T16:43:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.053+0000 7f3543e00640 1 Processor -- start 2024-09-13T16:43:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.054+0000 7f3543e00640 1 -- start start 2024-09-13T16:43:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.054+0000 7f3543e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c0c1ea0 0x7f353c152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.054+0000 7f3543e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f353c0c3670 0x7f353c1edf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.054+0000 7f3543e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f353c0c5bd0 0x7f353c157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.054+0000 7f3543e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353c0c0f80 con 0x7f353c0c1ea0 2024-09-13T16:43:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.054+0000 7f3543e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353c1ee4d0 con 0x7f353c0c3670 2024-09-13T16:43:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.054+0000 7f3543e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f353c151ed0 con 0x7f353c0c5bd0 2024-09-13T16:43:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.055+0000 7f3541400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c0c1ea0 0x7f353c152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.055+0000 7f3541400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c0c1ea0 0x7f353c152360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47338/0 (socket says 172.21.15.28:47338) 2024-09-13T16:43:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.055+0000 7f3541400640 1 -- 172.21.15.28:0/172127188 learned_addr learned my addr 172.21.15.28:0/172127188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.055+0000 7f3540a00640 1 --2- 172.21.15.28:0/172127188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f353c0c3670 0x7f353c1edf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.055+0000 7f3541e00640 1 --2- 172.21.15.28:0/172127188 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f353c0c5bd0 0x7f353c157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.055+0000 7f3541400640 1 -- 172.21.15.28:0/172127188 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f353c0c5bd0 msgr2=0x7f353c157c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.055+0000 7f3541400640 1 --2- 172.21.15.28:0/172127188 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f353c0c5bd0 0x7f353c157c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.055+0000 7f3541400640 1 -- 172.21.15.28:0/172127188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f353c0c3670 msgr2=0x7f353c1edf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.055+0000 7f3541400640 1 --2- 172.21.15.28:0/172127188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f353c0c3670 0x7f353c1edf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.055+0000 7f3541400640 1 -- 172.21.15.28:0/172127188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3528009400 con 0x7f353c0c1ea0 2024-09-13T16:43:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.055+0000 7f3541400640 1 --2- 172.21.15.28:0/172127188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c0c1ea0 0x7f353c152360 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f353000c570 tx=0x7f353000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.056+0000 7f3539600640 1 -- 172.21.15.28:0/172127188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3530005480 con 0x7f353c0c1ea0 2024-09-13T16:43:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.056+0000 7f3539600640 1 -- 172.21.15.28:0/172127188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35300035f0 con 0x7f353c0c1ea0 2024-09-13T16:43:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.056+0000 7f3539600640 1 -- 172.21.15.28:0/172127188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35300039a0 con 0x7f353c0c1ea0 2024-09-13T16:43:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.056+0000 7f3543e00640 1 -- 172.21.15.28:0/172127188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f353c0c0ce0 con 0x7f353c0c1ea0 2024-09-13T16:43:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.056+0000 7f3543e00640 1 -- 172.21.15.28:0/172127188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f353c151120 con 0x7f353c0c1ea0 2024-09-13T16:43:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.057+0000 7f3543e00640 1 -- 172.21.15.28:0/172127188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f353c069e80 con 0x7f353c0c1ea0 2024-09-13T16:43:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.058+0000 7f3539600640 1 -- 172.21.15.28:0/172127188 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7f3530003b00 con 0x7f353c0c1ea0 2024-09-13T16:43:51.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.059+0000 7f3539600640 1 --2- 172.21.15.28:0/172127188 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f351c06ab70 0x7f351c06d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:51.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.059+0000 7f3539600640 1 -- 172.21.15.28:0/172127188 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4846+0+0 (secure 0 0 0) 0x7f3530088580 con 0x7f353c0c1ea0 2024-09-13T16:43:51.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.059+0000 7f3540a00640 1 --2- 172.21.15.28:0/172127188 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f351c06ab70 0x7f351c06d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:51.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.060+0000 7f3540a00640 1 --2- 172.21.15.28:0/172127188 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f351c06ab70 0x7f351c06d030 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3528034000 tx=0x7f352802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:51.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.064+0000 7f3539600640 1 -- 172.21.15.28:0/172127188 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3530053bc0 con 0x7f353c0c1ea0 2024-09-13T16:43:51.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.281+0000 7f3543e00640 1 -- 172.21.15.28:0/172127188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f353c071800 con 0x7f353c0c1ea0 2024-09-13T16:43:51.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.282+0000 7f3539600640 1 -- 172.21.15.28:0/172127188 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5039 (secure 0 0 0) 0x7f3530017070 con 0x7f353c0c1ea0 2024-09-13T16:43:51.284 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:51.284 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4227,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/355570369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":355570369},{"type":"v1","addr":"172.21.15.28:6835","nonce":355570369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4232,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2366383263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6837","nonce":2366383263},{"type":"v1","addr":"172.21.15.191:6839","nonce":2366383263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6838/2702860767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2702860767},{"type":"v1","addr":"172.21.15.191:6838","nonce":2702860767}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2240589617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2240589617},{"type":"v1","addr":"172.21.15.28:6833","nonce":2240589617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/4068563017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":4068563017},{"type":"v1","addr":"172.21.15.28:6837","nonce":4068563017}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/704943585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":704943585},{"type":"v1","addr":"172.21.15.191:6835","nonce":704943585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2024-09-13T16:43:51.284 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-09-13T16:43:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.284+0000 7f3543e00640 1 -- 172.21.15.28:0/172127188 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f351c06ab70 msgr2=0x7f351c06d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.284+0000 7f3543e00640 1 --2- 172.21.15.28:0/172127188 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f351c06ab70 0x7f351c06d030 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3528034000 tx=0x7f352802e000 comp rx=0 tx=0).stop 2024-09-13T16:43:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.284+0000 7f3543e00640 1 -- 172.21.15.28:0/172127188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c0c1ea0 msgr2=0x7f353c152360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.284+0000 7f3543e00640 1 --2- 172.21.15.28:0/172127188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c0c1ea0 0x7f353c152360 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f353000c570 tx=0x7f353000ca30 comp rx=0 tx=0).stop 2024-09-13T16:43:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.284+0000 7f3543e00640 1 -- 172.21.15.28:0/172127188 shutdown_connections 2024-09-13T16:43:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.284+0000 7f3543e00640 1 --2- 172.21.15.28:0/172127188 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f351c06ab70 0x7f351c06d030 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.284+0000 7f3543e00640 1 --2- 172.21.15.28:0/172127188 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f353c0c5bd0 0x7f353c157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.284+0000 7f3543e00640 1 --2- 172.21.15.28:0/172127188 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f353c0c3670 0x7f353c1edf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.284+0000 7f3543e00640 1 --2- 172.21.15.28:0/172127188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c0c1ea0 0x7f353c152360 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.284+0000 7f3543e00640 1 -- 172.21.15.28:0/172127188 >> 172.21.15.28:0/172127188 conn(0x7f353c066040 msgr2=0x7f353c153f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.284+0000 7f3543e00640 1 -- 172.21.15.28:0/172127188 shutdown_connections 2024-09-13T16:43:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.284+0000 7f3543e00640 1 -- 172.21.15.28:0/172127188 wait complete. 2024-09-13T16:43:51.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:43:51.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.454+0000 7efc2f000640 1 Processor -- start 2024-09-13T16:43:51.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.454+0000 7efc2f000640 1 -- start start 2024-09-13T16:43:51.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.455+0000 7efc2f000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc280c5bd0 0x7efc28156150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:51.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.455+0000 7efc2f000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7efc280c6ef0 0x7efc280c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:51.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.455+0000 7efc2f000640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc2814ca00 con 0x7efc280c3670 2024-09-13T16:43:51.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.455+0000 7efc2f000640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc280c2dc0 con 0x7efc280c6ef0 2024-09-13T16:43:51.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.455+0000 7efc2f000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc28155780 con 0x7efc280c5bd0 2024-09-13T16:43:51.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.456+0000 7efc27e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc280c5bd0 0x7efc28156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:51.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.456+0000 7efc27e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc280c5bd0 0x7efc28156150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:57488/0 (socket says 172.21.15.28:57488) 2024-09-13T16:43:51.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.456+0000 7efc27e00640 1 -- 172.21.15.28:0/593337749 learned_addr learned my addr 172.21.15.28:0/593337749 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:51.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.456+0000 7efc27400640 1 --1- 172.21.15.28:0/593337749 >> v1:172.21.15.191:6789/0 conn(0x7efc280c3670 0x7efc280c1960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:46952/0 (socket says 172.21.15.28:46952) 2024-09-13T16:43:51.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.456+0000 7efc2d000640 1 --2- 172.21.15.28:0/593337749 >> v2:172.21.15.191:3300/0 conn(0x7efc280c6ef0 0x7efc280c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:51.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.456+0000 7efc27e00640 1 -- 172.21.15.28:0/593337749 >> v2:172.21.15.191:3300/0 conn(0x7efc280c6ef0 msgr2=0x7efc280c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:51.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.456+0000 7efc27e00640 1 --2- 172.21.15.28:0/593337749 >> v2:172.21.15.191:3300/0 conn(0x7efc280c6ef0 0x7efc280c7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.456+0000 7efc27e00640 1 -- 172.21.15.28:0/593337749 >> v1:172.21.15.191:6789/0 conn(0x7efc280c3670 legacy=0x7efc280c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:51.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.456+0000 7efc27e00640 1 -- 172.21.15.28:0/593337749 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc28155990 con 0x7efc280c5bd0 2024-09-13T16:43:51.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.456+0000 7efc26a00640 1 -- 172.21.15.28:0/593337749 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089940742 0 0) 0x7efc2814ca00 con 0x7efc280c3670 2024-09-13T16:43:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.457+0000 7efc27e00640 1 --2- 172.21.15.28:0/593337749 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc280c5bd0 0x7efc28156150 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efc18009870 tx=0x7efc1802a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6e91edfaa335c28 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.457+0000 7efc26a00640 1 -- 172.21.15.28:0/593337749 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc18032640 con 0x7efc280c5bd0 2024-09-13T16:43:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.457+0000 7efc26a00640 1 -- 172.21.15.28:0/593337749 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc180327a0 con 0x7efc280c5bd0 2024-09-13T16:43:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.457+0000 7efc26a00640 1 -- 172.21.15.28:0/593337749 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc18032b30 con 0x7efc280c5bd0 2024-09-13T16:43:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.458+0000 7efc2f000640 1 -- 172.21.15.28:0/593337749 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc280c5bd0 msgr2=0x7efc28156150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.458+0000 7efc2f000640 1 --2- 172.21.15.28:0/593337749 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc280c5bd0 0x7efc28156150 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efc18009870 tx=0x7efc1802a300 comp rx=0 tx=0).stop 2024-09-13T16:43:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.458+0000 7efc2f000640 1 -- 172.21.15.28:0/593337749 shutdown_connections 2024-09-13T16:43:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.458+0000 7efc2f000640 1 --2- 172.21.15.28:0/593337749 >> v2:172.21.15.191:3300/0 conn(0x7efc280c6ef0 0x7efc280c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.458+0000 7efc2f000640 1 --2- 172.21.15.28:0/593337749 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc280c5bd0 0x7efc28156150 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.458+0000 7efc2f000640 1 -- 172.21.15.28:0/593337749 >> 172.21.15.28:0/593337749 conn(0x7efc28066040 msgr2=0x7efc28069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.458+0000 7efc2f000640 1 -- 172.21.15.28:0/593337749 shutdown_connections 2024-09-13T16:43:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.458+0000 7efc2f000640 1 -- 172.21.15.28:0/593337749 wait complete. 2024-09-13T16:43:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.460+0000 7efc2f000640 1 Processor -- start 2024-09-13T16:43:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.460+0000 7efc2f000640 1 -- start start 2024-09-13T16:43:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.461+0000 7efc2f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc280c3670 0x7efc2814c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.461+0000 7efc2f000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc280c5bd0 0x7efc281e1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.461+0000 7efc2f000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc280c6ef0 0x7efc281e24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.461+0000 7efc2f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc281512c0 con 0x7efc280c3670 2024-09-13T16:43:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.461+0000 7efc2f000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc281ee960 con 0x7efc280c5bd0 2024-09-13T16:43:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.461+0000 7efc2f000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc281eec00 con 0x7efc280c6ef0 2024-09-13T16:43:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.461+0000 7efc27e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc280c3670 0x7efc2814c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.461+0000 7efc27400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc280c5bd0 0x7efc281e1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.461+0000 7efc27e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc280c3670 0x7efc2814c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47354/0 (socket says 172.21.15.28:47354) 2024-09-13T16:43:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.461+0000 7efc27e00640 1 -- 172.21.15.28:0/3108627607 learned_addr learned my addr 172.21.15.28:0/3108627607 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.461+0000 7efc2d000640 1 --2- 172.21.15.28:0/3108627607 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc280c6ef0 0x7efc281e24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.461+0000 7efc27e00640 1 -- 172.21.15.28:0/3108627607 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc280c6ef0 msgr2=0x7efc281e24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.461+0000 7efc27e00640 1 --2- 172.21.15.28:0/3108627607 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc280c6ef0 0x7efc281e24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.462+0000 7efc27e00640 1 -- 172.21.15.28:0/3108627607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc280c5bd0 msgr2=0x7efc281e1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:51.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.462+0000 7efc27e00640 1 --2- 172.21.15.28:0/3108627607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc280c5bd0 0x7efc281e1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.462+0000 7efc27e00640 1 -- 172.21.15.28:0/3108627607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc180094d0 con 0x7efc280c3670 2024-09-13T16:43:51.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.462+0000 7efc27400640 1 --2- 172.21.15.28:0/3108627607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc280c5bd0 0x7efc281e1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:51.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.462+0000 7efc27e00640 1 --2- 172.21.15.28:0/3108627607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc280c3670 0x7efc2814c0a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efc18009870 tx=0x7efc180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:51.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.462+0000 7efc1fe00640 1 -- 172.21.15.28:0/3108627607 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc18004230 con 0x7efc280c3670 2024-09-13T16:43:51.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.462+0000 7efc1fe00640 1 -- 172.21.15.28:0/3108627607 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc18004390 con 0x7efc280c3670 2024-09-13T16:43:51.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.462+0000 7efc1fe00640 1 -- 172.21.15.28:0/3108627607 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1803e960 con 0x7efc280c3670 2024-09-13T16:43:51.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.462+0000 7efc2f000640 1 -- 172.21.15.28:0/3108627607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc281eeea0 con 0x7efc280c3670 2024-09-13T16:43:51.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.462+0000 7efc2f000640 1 -- 172.21.15.28:0/3108627607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc2814c5e0 con 0x7efc280c3670 2024-09-13T16:43:51.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.463+0000 7efc2f000640 1 -- 172.21.15.28:0/3108627607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc28155c30 con 0x7efc280c3670 2024-09-13T16:43:51.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.464+0000 7efc1fe00640 1 -- 172.21.15.28:0/3108627607 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7efc1803eac0 con 0x7efc280c3670 2024-09-13T16:43:51.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.465+0000 7efc1fe00640 1 --2- 172.21.15.28:0/3108627607 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7efc0806ab70 0x7efc0806d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:51.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.465+0000 7efc1fe00640 1 -- 172.21.15.28:0/3108627607 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4846+0+0 (secure 0 0 0) 0x7efc180aa180 con 0x7efc280c3670 2024-09-13T16:43:51.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.465+0000 7efc27400640 1 --2- 172.21.15.28:0/3108627607 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7efc0806ab70 0x7efc0806d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:51.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.466+0000 7efc27400640 1 --2- 172.21.15.28:0/3108627607 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7efc0806ab70 0x7efc0806d030 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efc14000f60 tx=0x7efc14007d50 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:51.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.469+0000 7efc1fe00640 1 -- 172.21.15.28:0/3108627607 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7efc18075960 con 0x7efc280c3670 2024-09-13T16:43:51.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.651+0000 7efc2f000640 1 -- 172.21.15.28:0/3108627607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc2806e220 con 0x7efc280c3670 2024-09-13T16:43:51.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.651+0000 7efc1fe00640 1 -- 172.21.15.28:0/3108627607 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+12839 (secure 0 0 0) 0x7efc18078f80 con 0x7efc280c3670 2024-09-13T16:43:51.653 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:51.653 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":15,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:43:51.502767+0000","last_up_change":"2024-09-13T16:43:49.476313+0000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:43:49.643606+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6801","nonce":2964323435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6803","nonce":2964323435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6807","nonce":2964323435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6805","nonce":2964323435}]},"public_addr":"172.21.15.28:6801/2964323435","cluster_addr":"172.21.15.28:6803/2964323435","heartbeat_back_addr":"172.21.15.28:6807/2964323435","heartbeat_front_addr":"172.21.15.28:6805/2964323435","state":["exists","up"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6809","nonce":23768570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6811","nonce":23768570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6815","nonce":23768570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6813","nonce":23768570}]},"public_addr":"172.21.15.28:6809/23768570","cluster_addr":"172.21.15.28:6811/23768570","heartbeat_back_addr":"172.21.15.28:6815/23768570","heartbeat_front_addr":"172.21.15.28:6813/23768570","state":["exists","up"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6817","nonce":886887303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6819","nonce":886887303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6823","nonce":886887303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6821","nonce":886887303}]},"public_addr":"172.21.15.28:6817/886887303","cluster_addr":"172.21.15.28:6819/886887303","heartbeat_back_addr":"172.21.15.28:6823/886887303","heartbeat_front_addr":"172.21.15.28:6821/886887303","state":["exists","up"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6825","nonce":2616036674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6827","nonce":2616036674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6831","nonce":2616036674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6829","nonce":2616036674}]},"public_addr":"172.21.15.28:6825/2616036674","cluster_addr":"172.21.15.28:6827/2616036674","heartbeat_back_addr":"172.21.15.28:6831/2616036674","heartbeat_front_addr":"172.21.15.28:6829/2616036674","state":["exists","up"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6801","nonce":2817102450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6803","nonce":2817102450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6807","nonce":2817102450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6805","nonce":2817102450}]},"public_addr":"172.21.15.191:6801/2817102450","cluster_addr":"172.21.15.191:6803/2817102450","heartbeat_back_addr":"172.21.15.191:6807/2817102450","heartbeat_front_addr":"172.21.15.191:6805/2817102450","state":["exists","up"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6817","nonce":2503027971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6819","nonce":2503027971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6823","nonce":2503027971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6821","nonce":2503027971}]},"public_addr":"172.21.15.191:6817/2503027971","cluster_addr":"172.21.15.191:6819/2503027971","heartbeat_back_addr":"172.21.15.191:6823/2503027971","heartbeat_front_addr":"172.21.15.191:6821/2503027971","state":["exists","up"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6809","nonce":3006595088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6811","nonce":3006595088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6815","nonce":3006595088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6813","nonce":3006595088}]},"public_addr":"172.21.15.191:6809/3006595088","cluster_addr":"172.21.15.191:6811/3006595088","heartbeat_back_addr":"172.21.15.191:6815/3006595088","heartbeat_front_addr":"172.21.15.191:6813/3006595088","state":["exists","up"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6825","nonce":1477327720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6827","nonce":1477327720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6831","nonce":1477327720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6829","nonce":1477327720}]},"public_addr":"172.21.15.191:6825/1477327720","cluster_addr":"172.21.15.191:6827/1477327720","heartbeat_back_addr":"172.21.15.191:6831/1477327720","heartbeat_front_addr":"172.21.15.191:6829/1477327720","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.917250+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.954351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.017329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.046221+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.707467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.726947+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.782321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.764850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:43:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.654+0000 7efc2f000640 1 -- 172.21.15.28:0/3108627607 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7efc0806ab70 msgr2=0x7efc0806d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.654+0000 7efc2f000640 1 --2- 172.21.15.28:0/3108627607 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7efc0806ab70 0x7efc0806d030 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efc14000f60 tx=0x7efc14007d50 comp rx=0 tx=0).stop 2024-09-13T16:43:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.654+0000 7efc2f000640 1 -- 172.21.15.28:0/3108627607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc280c3670 msgr2=0x7efc2814c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.654+0000 7efc2f000640 1 --2- 172.21.15.28:0/3108627607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc280c3670 0x7efc2814c0a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efc18009870 tx=0x7efc180046c0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.654+0000 7efc2f000640 1 -- 172.21.15.28:0/3108627607 shutdown_connections 2024-09-13T16:43:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.654+0000 7efc2f000640 1 --2- 172.21.15.28:0/3108627607 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7efc0806ab70 0x7efc0806d030 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.654+0000 7efc2f000640 1 --2- 172.21.15.28:0/3108627607 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc280c6ef0 0x7efc281e24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.654+0000 7efc2f000640 1 --2- 172.21.15.28:0/3108627607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc280c5bd0 0x7efc281e1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.654+0000 7efc2f000640 1 --2- 172.21.15.28:0/3108627607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc280c3670 0x7efc2814c0a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.654+0000 7efc2f000640 1 -- 172.21.15.28:0/3108627607 >> 172.21.15.28:0/3108627607 conn(0x7efc28066040 msgr2=0x7efc28070af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.654+0000 7efc2f000640 1 -- 172.21.15.28:0/3108627607 shutdown_connections 2024-09-13T16:43:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.654+0000 7efc2f000640 1 -- 172.21.15.28:0/3108627607 wait complete. 2024-09-13T16:43:51.670 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:43:49.643606+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:43:51.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:43:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.826+0000 7fc1a4000640 1 Processor -- start 2024-09-13T16:43:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.826+0000 7fc1a4000640 1 -- start start 2024-09-13T16:43:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.827+0000 7fc1a4000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc19c0cafb0 0x7fc19c0c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.827+0000 7fc1a4000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc19c0c9970 0x7fc19c1610f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.827+0000 7fc1a4000640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c06e7f0 con 0x7fc19c150b30 2024-09-13T16:43:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.827+0000 7fc1a4000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc19c0c5ac0 con 0x7fc19c0c9970 2024-09-13T16:43:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.827+0000 7fc1a4000640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc19c06e570 con 0x7fc19c0cafb0 2024-09-13T16:43:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.828+0000 7fc1a2000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc19c0c9970 0x7fc19c1610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.828+0000 7fc1a2000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc19c0c9970 0x7fc19c1610f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47362/0 (socket says 172.21.15.28:47362) 2024-09-13T16:43:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.828+0000 7fc1a2000640 1 -- 172.21.15.28:0/565137287 learned_addr learned my addr 172.21.15.28:0/565137287 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.828+0000 7fc1a0c00640 1 --2- 172.21.15.28:0/565137287 >> v2:172.21.15.191:3300/0 conn(0x7fc19c0cafb0 0x7fc19c0c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.828+0000 7fc1a2000640 1 -- 172.21.15.28:0/565137287 >> v2:172.21.15.191:3300/0 conn(0x7fc19c0cafb0 msgr2=0x7fc19c0c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.828+0000 7fc1a2000640 1 --2- 172.21.15.28:0/565137287 >> v2:172.21.15.191:3300/0 conn(0x7fc19c0cafb0 0x7fc19c0c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.828+0000 7fc1a2000640 1 -- 172.21.15.28:0/565137287 >> v1:172.21.15.191:6789/0 conn(0x7fc19c150b30 legacy=0x7fc19c0c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.828+0000 7fc1a2000640 1 -- 172.21.15.28:0/565137287 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c0c9de0 con 0x7fc19c0c9970 2024-09-13T16:43:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.829+0000 7fc1a2000640 1 --2- 172.21.15.28:0/565137287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc19c0c9970 0x7fc19c1610f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc19c06fb00 tx=0x7fc19402a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a35267dc646962 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.829+0000 7fc19be00640 1 -- 172.21.15.28:0/565137287 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc194031600 con 0x7fc19c0c9970 2024-09-13T16:43:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.829+0000 7fc19be00640 1 -- 172.21.15.28:0/565137287 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19402d050 con 0x7fc19c0c9970 2024-09-13T16:43:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.829+0000 7fc19be00640 1 -- 172.21.15.28:0/565137287 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1940319a0 con 0x7fc19c0c9970 2024-09-13T16:43:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.830+0000 7fc1a4000640 1 -- 172.21.15.28:0/565137287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc19c0c9970 msgr2=0x7fc19c1610f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.830+0000 7fc1a4000640 1 --2- 172.21.15.28:0/565137287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc19c0c9970 0x7fc19c1610f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc19c06fb00 tx=0x7fc19402a7c0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.830+0000 7fc1a4000640 1 -- 172.21.15.28:0/565137287 shutdown_connections 2024-09-13T16:43:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.830+0000 7fc1a4000640 1 --2- 172.21.15.28:0/565137287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc19c0c9970 0x7fc19c1610f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.830+0000 7fc1a4000640 1 --2- 172.21.15.28:0/565137287 >> v2:172.21.15.191:3300/0 conn(0x7fc19c0cafb0 0x7fc19c0c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.830+0000 7fc1a4000640 1 -- 172.21.15.28:0/565137287 >> 172.21.15.28:0/565137287 conn(0x7fc19c0ca2f0 msgr2=0x7fc19c0ca6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.830+0000 7fc1a4000640 1 -- 172.21.15.28:0/565137287 shutdown_connections 2024-09-13T16:43:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.830+0000 7fc1a4000640 1 -- 172.21.15.28:0/565137287 wait complete. 2024-09-13T16:43:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.832+0000 7fc1a4000640 1 Processor -- start 2024-09-13T16:43:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.832+0000 7fc1a4000640 1 -- start start 2024-09-13T16:43:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.833+0000 7fc1a4000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc19c0c9970 0x7fc19c1e40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.833+0000 7fc1a4000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc19c0cafb0 0x7fc19c143330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.833+0000 7fc1a4000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc19c150b30 0x7fc19c140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.833+0000 7fc1a4000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c140ab0 con 0x7fc19c0c9970 2024-09-13T16:43:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.833+0000 7fc1a4000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c140480 con 0x7fc19c0cafb0 2024-09-13T16:43:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.833+0000 7fc1a4000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c1e3af0 con 0x7fc19c150b30 2024-09-13T16:43:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.833+0000 7fc1a1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc19c0c9970 0x7fc19c1e40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.833+0000 7fc1a1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc19c0c9970 0x7fc19c1e40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47364/0 (socket says 172.21.15.28:47364) 2024-09-13T16:43:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.833+0000 7fc1a1600640 1 -- 172.21.15.28:0/535984985 learned_addr learned my addr 172.21.15.28:0/535984985 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.833+0000 7fc1a0c00640 1 --2- 172.21.15.28:0/535984985 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc19c0cafb0 0x7fc19c143330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.833+0000 7fc1a2000640 1 --2- 172.21.15.28:0/535984985 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc19c150b30 0x7fc19c140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.833+0000 7fc1a0c00640 1 -- 172.21.15.28:0/535984985 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc19c150b30 msgr2=0x7fc19c140ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.833+0000 7fc1a0c00640 1 --2- 172.21.15.28:0/535984985 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc19c150b30 0x7fc19c140ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.833+0000 7fc1a0c00640 1 -- 172.21.15.28:0/535984985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc19c0c9970 msgr2=0x7fc19c1e40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.833+0000 7fc1a0c00640 1 --2- 172.21.15.28:0/535984985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc19c0c9970 0x7fc19c1e40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.833+0000 7fc1a0c00640 1 -- 172.21.15.28:0/535984985 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc194009b30 con 0x7fc19c0cafb0 2024-09-13T16:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.833+0000 7fc1a1600640 1 --2- 172.21.15.28:0/535984985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc19c0c9970 0x7fc19c1e40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.834+0000 7fc1a0c00640 1 --2- 172.21.15.28:0/535984985 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc19c0cafb0 0x7fc19c143330 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc18800e820 tx=0x7fc18800ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.834+0000 7fc199600640 1 -- 172.21.15.28:0/535984985 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1880095b0 con 0x7fc19c0cafb0 2024-09-13T16:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.834+0000 7fc199600640 1 -- 172.21.15.28:0/535984985 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18800c940 con 0x7fc19c0cafb0 2024-09-13T16:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.834+0000 7fc199600640 1 -- 172.21.15.28:0/535984985 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18800ccf0 con 0x7fc19c0cafb0 2024-09-13T16:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.834+0000 7fc1a4000640 1 -- 172.21.15.28:0/535984985 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19c1e3dd0 con 0x7fc19c0cafb0 2024-09-13T16:43:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.834+0000 7fc1a4000640 1 -- 172.21.15.28:0/535984985 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19c1eeea0 con 0x7fc19c0cafb0 2024-09-13T16:43:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.835+0000 7fc1a4000640 1 -- 172.21.15.28:0/535984985 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19c140d20 con 0x7fc19c0cafb0 2024-09-13T16:43:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.836+0000 7fc199600640 1 -- 172.21.15.28:0/535984985 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7fc188013470 con 0x7fc19c0cafb0 2024-09-13T16:43:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.837+0000 7fc199600640 1 --2- 172.21.15.28:0/535984985 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc17c06ab70 0x7fc17c06d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.837+0000 7fc199600640 1 -- 172.21.15.28:0/535984985 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc188089420 con 0x7fc19c0cafb0 2024-09-13T16:43:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.837+0000 7fc1a1600640 1 --2- 172.21.15.28:0/535984985 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc17c06ab70 0x7fc17c06d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.838+0000 7fc1a1600640 1 --2- 172.21.15.28:0/535984985 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc17c06ab70 0x7fc17c06d030 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc190007fd0 tx=0x7fc190009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:51.842+0000 7fc199600640 1 -- 172.21.15.28:0/535984985 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc188054a40 con 0x7fc19c0cafb0 2024-09-13T16:43:52.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.027+0000 7fc1a4000640 1 -- 172.21.15.28:0/535984985 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc19c0c5350 con 0x7fc19c0cafb0 2024-09-13T16:43:52.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.027+0000 7fc199600640 1 -- 172.21.15.28:0/535984985 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v15) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc1880170a0 con 0x7fc19c0cafb0 2024-09-13T16:43:52.029 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T16:43:52.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.029+0000 7fc1a4000640 1 -- 172.21.15.28:0/535984985 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc17c06ab70 msgr2=0x7fc17c06d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:52.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.029+0000 7fc1a4000640 1 --2- 172.21.15.28:0/535984985 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc17c06ab70 0x7fc17c06d030 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc190007fd0 tx=0x7fc190009040 comp rx=0 tx=0).stop 2024-09-13T16:43:52.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.029+0000 7fc1a4000640 1 -- 172.21.15.28:0/535984985 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc19c0cafb0 msgr2=0x7fc19c143330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:52.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.029+0000 7fc1a4000640 1 --2- 172.21.15.28:0/535984985 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc19c0cafb0 0x7fc19c143330 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc18800e820 tx=0x7fc18800ece0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.030+0000 7fc1a4000640 1 -- 172.21.15.28:0/535984985 shutdown_connections 2024-09-13T16:43:52.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.030+0000 7fc1a4000640 1 --2- 172.21.15.28:0/535984985 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc17c06ab70 0x7fc17c06d030 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.030+0000 7fc1a4000640 1 --2- 172.21.15.28:0/535984985 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc19c150b30 0x7fc19c140ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.030+0000 7fc1a4000640 1 --2- 172.21.15.28:0/535984985 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc19c0cafb0 0x7fc19c143330 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.030+0000 7fc1a4000640 1 --2- 172.21.15.28:0/535984985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc19c0c9970 0x7fc19c1e40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.030+0000 7fc1a4000640 1 -- 172.21.15.28:0/535984985 >> 172.21.15.28:0/535984985 conn(0x7fc19c0ca2f0 msgr2=0x7fc19c0c80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:52.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.030+0000 7fc1a4000640 1 -- 172.21.15.28:0/535984985 shutdown_connections 2024-09-13T16:43:52.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.030+0000 7fc1a4000640 1 -- 172.21.15.28:0/535984985 wait complete. 2024-09-13T16:43:52.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T16:43:52.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.201+0000 7fea31400640 1 Processor -- start 2024-09-13T16:43:52.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.201+0000 7fea31400640 1 -- start start 2024-09-13T16:43:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.202+0000 7fea31400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fea2c0c9be0 0x7fea2c0c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.202+0000 7fea31400640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fea2c152730 0x7fea2c152b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.202+0000 7fea31400640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea2c1530a0 con 0x7fea2c1588d0 2024-09-13T16:43:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.202+0000 7fea31400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea2c15be40 con 0x7fea2c0c9be0 2024-09-13T16:43:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.202+0000 7fea31400640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea2c15c9c0 con 0x7fea2c152730 2024-09-13T16:43:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.203+0000 7fea23e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fea2c152730 0x7fea2c152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.203+0000 7fea30a00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fea2c1588d0 0x7fea2c06a6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:46972/0 (socket says 172.21.15.28:46972) 2024-09-13T16:43:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.203+0000 7fea23e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fea2c152730 0x7fea2c152b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:53294/0 (socket says 172.21.15.28:53294) 2024-09-13T16:43:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.203+0000 7fea30a00640 1 -- 172.21.15.28:0/789702849 learned_addr learned my addr 172.21.15.28:0/789702849 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:43:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.203+0000 7fea2be00640 1 --2- 172.21.15.28:0/789702849 >> v2:172.21.15.28:3300/0 conn(0x7fea2c0c9be0 0x7fea2c0c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.203+0000 7fea23e00640 1 -- 172.21.15.28:0/789702849 >> v2:172.21.15.28:3300/0 conn(0x7fea2c0c9be0 msgr2=0x7fea2c0c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.203+0000 7fea23e00640 1 --2- 172.21.15.28:0/789702849 >> v2:172.21.15.28:3300/0 conn(0x7fea2c0c9be0 0x7fea2c0c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.203+0000 7fea23e00640 1 -- 172.21.15.28:0/789702849 >> v1:172.21.15.191:6789/0 conn(0x7fea2c1588d0 legacy=0x7fea2c06a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.203+0000 7fea23e00640 1 -- 172.21.15.28:0/789702849 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea2c0ca500 con 0x7fea2c152730 2024-09-13T16:43:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.203+0000 7fea2b400640 1 -- 172.21.15.28:0/789702849 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236529377 0 0) 0x7fea24001490 con 0x7fea2c1588d0 2024-09-13T16:43:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.204+0000 7fea23e00640 1 --2- 172.21.15.28:0/789702849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea2c152730 0x7fea2c152b60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fea2c0c8ff0 tx=0x7fea1402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8afc1c9d2fb2d06 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.204+0000 7fea2b400640 1 -- 172.21.15.28:0/789702849 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14032440 con 0x7fea2c152730 2024-09-13T16:43:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.205+0000 7fea2b400640 1 -- 172.21.15.28:0/789702849 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea140325a0 con 0x7fea2c152730 2024-09-13T16:43:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.205+0000 7fea2b400640 1 -- 172.21.15.28:0/789702849 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14032930 con 0x7fea2c152730 2024-09-13T16:43:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.205+0000 7fea31400640 1 -- 172.21.15.28:0/789702849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea2c152730 msgr2=0x7fea2c152b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.205+0000 7fea31400640 1 --2- 172.21.15.28:0/789702849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea2c152730 0x7fea2c152b60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fea2c0c8ff0 tx=0x7fea1402a1a0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.205+0000 7fea31400640 1 -- 172.21.15.28:0/789702849 shutdown_connections 2024-09-13T16:43:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.205+0000 7fea31400640 1 --2- 172.21.15.28:0/789702849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea2c152730 0x7fea2c152b60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.205+0000 7fea31400640 1 --2- 172.21.15.28:0/789702849 >> v2:172.21.15.28:3300/0 conn(0x7fea2c0c9be0 0x7fea2c0c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.205+0000 7fea31400640 1 -- 172.21.15.28:0/789702849 >> 172.21.15.28:0/789702849 conn(0x7fea2c0719d0 msgr2=0x7fea2c065c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:52.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.205+0000 7fea31400640 1 -- 172.21.15.28:0/789702849 shutdown_connections 2024-09-13T16:43:52.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.205+0000 7fea31400640 1 -- 172.21.15.28:0/789702849 wait complete. 2024-09-13T16:43:52.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.207+0000 7fea31400640 1 Processor -- start 2024-09-13T16:43:52.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.207+0000 7fea31400640 1 -- start start 2024-09-13T16:43:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.208+0000 7fea31400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea2c0c9be0 0x7fea2c14e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.208+0000 7fea31400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea2c152730 0x7fea2c14cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.208+0000 7fea31400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea2c1588d0 0x7fea2c0cd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.208+0000 7fea31400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c1f2820 con 0x7fea2c0c9be0 2024-09-13T16:43:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.208+0000 7fea31400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c1ee300 con 0x7fea2c1588d0 2024-09-13T16:43:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.208+0000 7fea31400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c1f2540 con 0x7fea2c152730 2024-09-13T16:43:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.209+0000 7fea2be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea2c0c9be0 0x7fea2c14e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.209+0000 7fea23e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea2c152730 0x7fea2c14cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:52.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.209+0000 7fea30a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea2c1588d0 0x7fea2c0cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:52.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.209+0000 7fea23e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea2c152730 0x7fea2c14cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:57510/0 (socket says 172.21.15.28:57510) 2024-09-13T16:43:52.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.209+0000 7fea23e00640 1 -- 172.21.15.28:0/1887539482 learned_addr learned my addr 172.21.15.28:0/1887539482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:52.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.209+0000 7fea23e00640 1 -- 172.21.15.28:0/1887539482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea2c1588d0 msgr2=0x7fea2c0cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:52.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.209+0000 7fea23e00640 1 --2- 172.21.15.28:0/1887539482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea2c1588d0 0x7fea2c0cd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.209+0000 7fea23e00640 1 -- 172.21.15.28:0/1887539482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea2c0c9be0 msgr2=0x7fea2c14e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:52.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.209+0000 7fea23e00640 1 --2- 172.21.15.28:0/1887539482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea2c0c9be0 0x7fea2c14e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.209+0000 7fea23e00640 1 -- 172.21.15.28:0/1887539482 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea14009400 con 0x7fea2c152730 2024-09-13T16:43:52.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.209+0000 7fea30a00640 1 --2- 172.21.15.28:0/1887539482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea2c1588d0 0x7fea2c0cd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:52.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.210+0000 7fea23e00640 1 --2- 172.21.15.28:0/1887539482 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea2c152730 0x7fea2c14cc40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fea14000c00 tx=0x7fea140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:52.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.210+0000 7fea28c00640 1 -- 172.21.15.28:0/1887539482 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14030b80 con 0x7fea2c152730 2024-09-13T16:43:52.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.210+0000 7fea28c00640 1 -- 172.21.15.28:0/1887539482 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea14032e90 con 0x7fea2c152730 2024-09-13T16:43:52.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.210+0000 7fea31400640 1 -- 172.21.15.28:0/1887539482 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea2c1ee510 con 0x7fea2c152730 2024-09-13T16:43:52.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.210+0000 7fea28c00640 1 -- 172.21.15.28:0/1887539482 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14006e10 con 0x7fea2c152730 2024-09-13T16:43:52.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.210+0000 7fea31400640 1 -- 172.21.15.28:0/1887539482 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea2c1eead0 con 0x7fea2c152730 2024-09-13T16:43:52.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.211+0000 7fea31400640 1 -- 172.21.15.28:0/1887539482 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea2c1eecf0 con 0x7fea2c152730 2024-09-13T16:43:52.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.213+0000 7fea28c00640 1 -- 172.21.15.28:0/1887539482 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7fea14039040 con 0x7fea2c152730 2024-09-13T16:43:52.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.214+0000 7fea28c00640 1 --2- 172.21.15.28:0/1887539482 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fea0806aba0 0x7fea0806d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:52.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.214+0000 7fea28c00640 1 -- 172.21.15.28:0/1887539482 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fea140a9840 con 0x7fea2c152730 2024-09-13T16:43:52.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.214+0000 7fea2be00640 1 --2- 172.21.15.28:0/1887539482 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fea0806aba0 0x7fea0806d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:52.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.215+0000 7fea2be00640 1 --2- 172.21.15.28:0/1887539482 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fea0806aba0 0x7fea0806d060 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fea1c009b40 tx=0x7fea1c009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.218+0000 7fea28c00640 1 -- 172.21.15.28:0/1887539482 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fea14074e60 con 0x7fea2c152730 2024-09-13T16:43:52.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.402+0000 7fea31400640 1 -- 172.21.15.28:0/1887539482 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fea2c0c4cf0 con 0x7fea2c152730 2024-09-13T16:43:52.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.403+0000 7fea28c00640 1 -- 172.21.15.28:0/1887539482 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v15) v1 ==== 84+0+65 (secure 0 0 0) 0x7fea14041090 con 0x7fea2c152730 2024-09-13T16:43:52.404 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:52.405 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T16:43:52.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:43:52.405 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-13T16:43:52.405 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-13T16:43:52.405 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:43:52.405 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T16:43:52.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.405+0000 7fea31400640 1 -- 172.21.15.28:0/1887539482 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fea0806aba0 msgr2=0x7fea0806d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:52.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.405+0000 7fea31400640 1 --2- 172.21.15.28:0/1887539482 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fea0806aba0 0x7fea0806d060 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fea1c009b40 tx=0x7fea1c009190 comp rx=0 tx=0).stop 2024-09-13T16:43:52.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.405+0000 7fea31400640 1 -- 172.21.15.28:0/1887539482 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea2c152730 msgr2=0x7fea2c14cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:52.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.405+0000 7fea31400640 1 --2- 172.21.15.28:0/1887539482 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea2c152730 0x7fea2c14cc40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fea14000c00 tx=0x7fea140046c0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.406+0000 7fea31400640 1 -- 172.21.15.28:0/1887539482 shutdown_connections 2024-09-13T16:43:52.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.406+0000 7fea31400640 1 --2- 172.21.15.28:0/1887539482 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fea0806aba0 0x7fea0806d060 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.406+0000 7fea31400640 1 --2- 172.21.15.28:0/1887539482 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea2c1588d0 0x7fea2c0cd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.406+0000 7fea31400640 1 --2- 172.21.15.28:0/1887539482 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea2c152730 0x7fea2c14cc40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.406+0000 7fea31400640 1 --2- 172.21.15.28:0/1887539482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea2c0c9be0 0x7fea2c14e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.406+0000 7fea31400640 1 -- 172.21.15.28:0/1887539482 >> 172.21.15.28:0/1887539482 conn(0x7fea2c0719d0 msgr2=0x7fea2c158d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:52.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.406+0000 7fea31400640 1 -- 172.21.15.28:0/1887539482 shutdown_connections 2024-09-13T16:43:52.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.406+0000 7fea31400640 1 -- 172.21.15.28:0/1887539482 wait complete. 2024-09-13T16:43:52.422 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T16:43:52.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T16:43:52.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.584+0000 7f35f1e00640 1 Processor -- start 2024-09-13T16:43:52.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.584+0000 7f35f1e00640 1 -- start start 2024-09-13T16:43:52.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.585+0000 7f35f1e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f35ec0c46f0 0x7f35ec0ca940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:52.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.585+0000 7f35f1e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ec0cbbe0 con 0x7f35ec159500 2024-09-13T16:43:52.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.585+0000 7f35f1e00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ec0cb060 con 0x7f35ec0c9940 2024-09-13T16:43:52.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.585+0000 7f35f1e00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ec0cae80 con 0x7f35ec0c46f0 2024-09-13T16:43:52.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.586+0000 7f35f0a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f35ec0c46f0 0x7f35ec0ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:52.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.586+0000 7f35f0a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f35ec0c46f0 0x7f35ec0ca940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:53310/0 (socket says 172.21.15.28:53310) 2024-09-13T16:43:52.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.586+0000 7f35f0a00640 1 -- 172.21.15.28:0/223221130 learned_addr learned my addr 172.21.15.28:0/223221130 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.586+0000 7f35f0a00640 1 -- 172.21.15.28:0/223221130 >> v1:172.21.15.191:6789/0 conn(0x7f35ec0c9940 legacy=0x7f35ec0c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.586+0000 7f35f0a00640 1 -- 172.21.15.28:0/223221130 >> v1:172.21.15.28:6789/0 conn(0x7f35ec159500 legacy=0x7f35ec0ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.586+0000 7f35f0a00640 1 -- 172.21.15.28:0/223221130 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec0cb340 con 0x7f35ec0c46f0 2024-09-13T16:43:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.587+0000 7f35f0a00640 1 --2- 172.21.15.28:0/223221130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35ec0c46f0 0x7f35ec0ca940 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f35e0009870 tx=0x7f35e002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40827de46c2558ac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:52.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.587+0000 7f35eb400640 1 -- 172.21.15.28:0/223221130 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e0032440 con 0x7f35ec0c46f0 2024-09-13T16:43:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.587+0000 7f35eb400640 1 -- 172.21.15.28:0/223221130 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e00325a0 con 0x7f35ec0c46f0 2024-09-13T16:43:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.587+0000 7f35eb400640 1 -- 172.21.15.28:0/223221130 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e0032930 con 0x7f35ec0c46f0 2024-09-13T16:43:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.588+0000 7f35f1e00640 1 -- 172.21.15.28:0/223221130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35ec0c46f0 msgr2=0x7f35ec0ca940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.588+0000 7f35f1e00640 1 --2- 172.21.15.28:0/223221130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35ec0c46f0 0x7f35ec0ca940 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f35e0009870 tx=0x7f35e002a270 comp rx=0 tx=0).stop 2024-09-13T16:43:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.588+0000 7f35f1e00640 1 -- 172.21.15.28:0/223221130 shutdown_connections 2024-09-13T16:43:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.588+0000 7f35f1e00640 1 --2- 172.21.15.28:0/223221130 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35ec0c46f0 0x7f35ec0ca940 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.588+0000 7f35f1e00640 1 -- 172.21.15.28:0/223221130 >> 172.21.15.28:0/223221130 conn(0x7f35ec06c830 msgr2=0x7f35ec06a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.588+0000 7f35f1e00640 1 -- 172.21.15.28:0/223221130 shutdown_connections 2024-09-13T16:43:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.588+0000 7f35f1e00640 1 -- 172.21.15.28:0/223221130 wait complete. 2024-09-13T16:43:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.590+0000 7f35f1e00640 1 Processor -- start 2024-09-13T16:43:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.590+0000 7f35f1e00640 1 -- start start 2024-09-13T16:43:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.591+0000 7f35f1e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35ec0c46f0 0x7f35ec150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.591+0000 7f35f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec0c9940 0x7f35ec14e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.591+0000 7f35f1e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35ec159500 0x7f35ec1574f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.591+0000 7f35f1e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec1570d0 con 0x7f35ec0c9940 2024-09-13T16:43:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.591+0000 7f35f1e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec1eeb00 con 0x7f35ec0c46f0 2024-09-13T16:43:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.591+0000 7f35f1e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec1eeda0 con 0x7f35ec159500 2024-09-13T16:43:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.591+0000 7f35ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec0c9940 0x7f35ec14e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.591+0000 7f35f0a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35ec0c46f0 0x7f35ec150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.591+0000 7f35ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec0c9940 0x7f35ec14e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47400/0 (socket says 172.21.15.28:47400) 2024-09-13T16:43:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.591+0000 7f35ebe00640 1 -- 172.21.15.28:0/2359620847 learned_addr learned my addr 172.21.15.28:0/2359620847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.591+0000 7f35f1400640 1 --2- 172.21.15.28:0/2359620847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35ec159500 0x7f35ec1574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.592+0000 7f35ebe00640 1 -- 172.21.15.28:0/2359620847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35ec159500 msgr2=0x7f35ec1574f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.592+0000 7f35ebe00640 1 --2- 172.21.15.28:0/2359620847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35ec159500 0x7f35ec1574f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.592+0000 7f35ebe00640 1 -- 172.21.15.28:0/2359620847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35ec0c46f0 msgr2=0x7f35ec150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.592+0000 7f35ebe00640 1 --2- 172.21.15.28:0/2359620847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35ec0c46f0 0x7f35ec150330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.592+0000 7f35ebe00640 1 -- 172.21.15.28:0/2359620847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e00094d0 con 0x7f35ec0c9940 2024-09-13T16:43:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.592+0000 7f35f0a00640 1 --2- 172.21.15.28:0/2359620847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35ec0c46f0 0x7f35ec150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.592+0000 7f35f1400640 1 --2- 172.21.15.28:0/2359620847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35ec159500 0x7f35ec1574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.592+0000 7f35ebe00640 1 --2- 172.21.15.28:0/2359620847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec0c9940 0x7f35ec14e990 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f35d800c450 tx=0x7f35d800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.592+0000 7f35e8c00640 1 -- 172.21.15.28:0/2359620847 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d80052d0 con 0x7f35ec0c9940 2024-09-13T16:43:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.592+0000 7f35e8c00640 1 -- 172.21.15.28:0/2359620847 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d8005430 con 0x7f35ec0c9940 2024-09-13T16:43:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.592+0000 7f35e8c00640 1 -- 172.21.15.28:0/2359620847 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d8012870 con 0x7f35ec0c9940 2024-09-13T16:43:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.592+0000 7f35f1e00640 1 -- 172.21.15.28:0/2359620847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35ec156730 con 0x7f35ec0c9940 2024-09-13T16:43:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.592+0000 7f35f1e00640 1 -- 172.21.15.28:0/2359620847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ec156cf0 con 0x7f35ec0c9940 2024-09-13T16:43:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.593+0000 7f35f1e00640 1 -- 172.21.15.28:0/2359620847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ec067460 con 0x7f35ec0c9940 2024-09-13T16:43:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.594+0000 7f35e8c00640 1 -- 172.21.15.28:0/2359620847 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7f35d80129d0 con 0x7f35ec0c9940 2024-09-13T16:43:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.595+0000 7f35e8c00640 1 --2- 172.21.15.28:0/2359620847 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f35cc06ab70 0x7f35cc06d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.595+0000 7f35e8c00640 1 -- 172.21.15.28:0/2359620847 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f35d80889c0 con 0x7f35ec0c9940 2024-09-13T16:43:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.596+0000 7f35f0a00640 1 --2- 172.21.15.28:0/2359620847 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f35cc06ab70 0x7f35cc06d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:52.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.596+0000 7f35f0a00640 1 --2- 172.21.15.28:0/2359620847 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f35cc06ab70 0x7f35cc06d030 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f35e0034000 tx=0x7f35e002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:52.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.600+0000 7f35e8c00640 1 -- 172.21.15.28:0/2359620847 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f35d8053fe0 con 0x7f35ec0c9940 2024-09-13T16:43:52.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:52.784+0000 7f35f1e00640 1 -- 172.21.15.28:0/2359620847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f35ec0b4090 con 0x7f35ec0c9940 2024-09-13T16:43:53.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.521+0000 7f35e8c00640 1 -- 172.21.15.28:0/2359620847 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f35d8057600 con 0x7f35ec0c9940 2024-09-13T16:43:53.522 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-13T16:43:53.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.524+0000 7f35f1e00640 1 -- 172.21.15.28:0/2359620847 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f35cc06ab70 msgr2=0x7f35cc06d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:53.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.524+0000 7f35f1e00640 1 --2- 172.21.15.28:0/2359620847 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f35cc06ab70 0x7f35cc06d030 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f35e0034000 tx=0x7f35e002e000 comp rx=0 tx=0).stop 2024-09-13T16:43:53.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.524+0000 7f35f1e00640 1 -- 172.21.15.28:0/2359620847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec0c9940 msgr2=0x7f35ec14e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:53.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.524+0000 7f35f1e00640 1 --2- 172.21.15.28:0/2359620847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec0c9940 0x7f35ec14e990 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f35d800c450 tx=0x7f35d800c910 comp rx=0 tx=0).stop 2024-09-13T16:43:53.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.525+0000 7f35f1e00640 1 -- 172.21.15.28:0/2359620847 shutdown_connections 2024-09-13T16:43:53.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.525+0000 7f35f1e00640 1 --2- 172.21.15.28:0/2359620847 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f35cc06ab70 0x7f35cc06d030 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:53.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.525+0000 7f35f1e00640 1 --2- 172.21.15.28:0/2359620847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35ec159500 0x7f35ec1574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:53.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.525+0000 7f35f1e00640 1 --2- 172.21.15.28:0/2359620847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec0c9940 0x7f35ec14e990 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f35d800c450 tx=0x7f35d800c910 comp rx=0 tx=0).stop 2024-09-13T16:43:53.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.525+0000 7f35f1e00640 1 --2- 172.21.15.28:0/2359620847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35ec0c46f0 0x7f35ec150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:53.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.525+0000 7f35f1e00640 1 -- 172.21.15.28:0/2359620847 >> 172.21.15.28:0/2359620847 conn(0x7f35ec06c830 msgr2=0x7f35ec0c6880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:53.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.525+0000 7f35f1e00640 1 -- 172.21.15.28:0/2359620847 shutdown_connections 2024-09-13T16:43:53.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.525+0000 7f35f1e00640 1 -- 172.21.15.28:0/2359620847 wait complete. 2024-09-13T16:43:53.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T16:43:53.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.707+0000 7fd669a00640 1 Processor -- start 2024-09-13T16:43:53.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.707+0000 7fd669a00640 1 -- start start 2024-09-13T16:43:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.708+0000 7fd669a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6640c1830 0x7fd66415bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.708+0000 7fd669a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd6640706b0 0x7fd66415c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.708+0000 7fd669a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66415cea0 con 0x7fd6640c4350 2024-09-13T16:43:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.708+0000 7fd669a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6640c2730 con 0x7fd6640706b0 2024-09-13T16:43:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.708+0000 7fd669a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66415cbc0 con 0x7fd6640c1830 2024-09-13T16:43:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.708+0000 7fd662a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd6640c4350 0x7fd66406be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34364/0 (socket says 172.21.15.28:34364) 2024-09-13T16:43:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.708+0000 7fd662a00640 1 -- 172.21.15.28:0/3256243783 learned_addr learned my addr 172.21.15.28:0/3256243783 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:43:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.708+0000 7fd663400640 1 --2- 172.21.15.28:0/3256243783 >> v2:172.21.15.28:3300/0 conn(0x7fd6640706b0 0x7fd66415c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.709+0000 7fd65a000640 1 --2- 172.21.15.28:0/3256243783 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6640c1830 0x7fd66415bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.709+0000 7fd662000640 1 -- 172.21.15.28:0/3256243783 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436209070 0 0) 0x7fd66415cea0 con 0x7fd6640c4350 2024-09-13T16:43:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.709+0000 7fd662000640 1 -- 172.21.15.28:0/3256243783 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd648003520 con 0x7fd6640c4350 2024-09-13T16:43:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.709+0000 7fd663400640 1 -- 172.21.15.28:0/3256243783 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6640c1830 msgr2=0x7fd66415bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.709+0000 7fd663400640 1 --2- 172.21.15.28:0/3256243783 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6640c1830 0x7fd66415bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.709+0000 7fd663400640 1 -- 172.21.15.28:0/3256243783 >> v1:172.21.15.28:6789/0 conn(0x7fd6640c4350 legacy=0x7fd66406be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.709+0000 7fd663400640 1 -- 172.21.15.28:0/3256243783 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd664071760 con 0x7fd6640706b0 2024-09-13T16:43:53.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.710+0000 7fd663400640 1 --2- 172.21.15.28:0/3256243783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6640706b0 0x7fd66415c4f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd65c009d90 tx=0x7fd65c02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf13d345c1151ce server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:53.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.710+0000 7fd662000640 1 -- 172.21.15.28:0/3256243783 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65c031600 con 0x7fd6640706b0 2024-09-13T16:43:53.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.710+0000 7fd662000640 1 -- 172.21.15.28:0/3256243783 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65c02d050 con 0x7fd6640706b0 2024-09-13T16:43:53.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.710+0000 7fd662000640 1 -- 172.21.15.28:0/3256243783 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65c031960 con 0x7fd6640706b0 2024-09-13T16:43:53.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.711+0000 7fd669a00640 1 -- 172.21.15.28:0/3256243783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6640706b0 msgr2=0x7fd66415c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:53.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.711+0000 7fd669a00640 1 --2- 172.21.15.28:0/3256243783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6640706b0 0x7fd66415c4f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd65c009d90 tx=0x7fd65c02a5d0 comp rx=0 tx=0).stop 2024-09-13T16:43:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.711+0000 7fd669a00640 1 -- 172.21.15.28:0/3256243783 shutdown_connections 2024-09-13T16:43:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.711+0000 7fd669a00640 1 --2- 172.21.15.28:0/3256243783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6640706b0 0x7fd66415c4f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.711+0000 7fd669a00640 1 --2- 172.21.15.28:0/3256243783 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6640c1830 0x7fd66415bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.711+0000 7fd669a00640 1 -- 172.21.15.28:0/3256243783 >> 172.21.15.28:0/3256243783 conn(0x7fd6640c7390 msgr2=0x7fd6640c77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.711+0000 7fd669a00640 1 -- 172.21.15.28:0/3256243783 shutdown_connections 2024-09-13T16:43:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.711+0000 7fd669a00640 1 -- 172.21.15.28:0/3256243783 wait complete. 2024-09-13T16:43:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.712+0000 7fd669a00640 1 Processor -- start 2024-09-13T16:43:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.712+0000 7fd669a00640 1 -- start start 2024-09-13T16:43:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.713+0000 7fd669a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6640706b0 0x7fd66415af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.713+0000 7fd669a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6640c1830 0x7fd66414d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.713+0000 7fd669a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6640c4350 0x7fd66414db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.713+0000 7fd669a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66414ef80 con 0x7fd6640706b0 2024-09-13T16:43:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.713+0000 7fd669a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66414c580 con 0x7fd6640c1830 2024-09-13T16:43:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.713+0000 7fd669a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66414d0b0 con 0x7fd6640c4350 2024-09-13T16:43:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.713+0000 7fd662a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6640706b0 0x7fd66415af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.713+0000 7fd662a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6640706b0 0x7fd66415af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47414/0 (socket says 172.21.15.28:47414) 2024-09-13T16:43:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.713+0000 7fd662a00640 1 -- 172.21.15.28:0/1572011704 learned_addr learned my addr 172.21.15.28:0/1572011704 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.714+0000 7fd663400640 1 --2- 172.21.15.28:0/1572011704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6640c4350 0x7fd66414db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.714+0000 7fd65a000640 1 --2- 172.21.15.28:0/1572011704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6640c1830 0x7fd66414d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.714+0000 7fd662a00640 1 -- 172.21.15.28:0/1572011704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6640c4350 msgr2=0x7fd66414db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.714+0000 7fd662a00640 1 --2- 172.21.15.28:0/1572011704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6640c4350 0x7fd66414db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.714+0000 7fd662a00640 1 -- 172.21.15.28:0/1572011704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6640c1830 msgr2=0x7fd66414d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.714+0000 7fd662a00640 1 --2- 172.21.15.28:0/1572011704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6640c1830 0x7fd66414d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.714+0000 7fd662a00640 1 -- 172.21.15.28:0/1572011704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65c009a70 con 0x7fd6640706b0 2024-09-13T16:43:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.714+0000 7fd662a00640 1 --2- 172.21.15.28:0/1572011704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6640706b0 0x7fd66415af50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd65400c430 tx=0x7fd65400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.714+0000 7fd65b400640 1 -- 172.21.15.28:0/1572011704 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd654005ce0 con 0x7fd6640706b0 2024-09-13T16:43:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.714+0000 7fd65b400640 1 -- 172.21.15.28:0/1572011704 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd654005e40 con 0x7fd6640706b0 2024-09-13T16:43:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.714+0000 7fd669a00640 1 -- 172.21.15.28:0/1572011704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6641f20a0 con 0x7fd6640706b0 2024-09-13T16:43:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.714+0000 7fd65b400640 1 -- 172.21.15.28:0/1572011704 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6540072f0 con 0x7fd6640706b0 2024-09-13T16:43:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.714+0000 7fd669a00640 1 -- 172.21.15.28:0/1572011704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66414c800 con 0x7fd6640706b0 2024-09-13T16:43:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.715+0000 7fd669a00640 1 -- 172.21.15.28:0/1572011704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6640c4730 con 0x7fd6640706b0 2024-09-13T16:43:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.717+0000 7fd65b400640 1 -- 172.21.15.28:0/1572011704 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7fd654007450 con 0x7fd6640706b0 2024-09-13T16:43:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.717+0000 7fd65b400640 1 --2- 172.21.15.28:0/1572011704 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd64006aba0 0x7fd64006d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.717+0000 7fd65b400640 1 -- 172.21.15.28:0/1572011704 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fd654088670 con 0x7fd6640706b0 2024-09-13T16:43:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.718+0000 7fd65a000640 1 --2- 172.21.15.28:0/1572011704 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd64006aba0 0x7fd64006d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.718+0000 7fd65a000640 1 --2- 172.21.15.28:0/1572011704 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd64006aba0 0x7fd64006d060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd6641e88d0 tx=0x7fd64c009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:53.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.722+0000 7fd65b400640 1 -- 172.21.15.28:0/1572011704 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd654053bb0 con 0x7fd6640706b0 2024-09-13T16:43:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:53.903+0000 7fd669a00640 1 -- 172.21.15.28:0/1572011704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd6640c8730 con 0x7fd6640706b0 2024-09-13T16:43:54.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.535+0000 7fd65b400640 1 -- 172.21.15.28:0/1572011704 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd654053f80 con 0x7fd6640706b0 2024-09-13T16:43:54.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.538+0000 7fd65b400640 1 -- 172.21.15.28:0/1572011704 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd6540736b0 con 0x7fd6640706b0 2024-09-13T16:43:54.540 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-13T16:43:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.544+0000 7fd669a00640 1 -- 172.21.15.28:0/1572011704 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd64006aba0 msgr2=0x7fd64006d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.544+0000 7fd669a00640 1 --2- 172.21.15.28:0/1572011704 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd64006aba0 0x7fd64006d060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd6641e88d0 tx=0x7fd64c009190 comp rx=0 tx=0).stop 2024-09-13T16:43:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.544+0000 7fd669a00640 1 -- 172.21.15.28:0/1572011704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6640706b0 msgr2=0x7fd66415af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.544+0000 7fd669a00640 1 --2- 172.21.15.28:0/1572011704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6640706b0 0x7fd66415af50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd65400c430 tx=0x7fd65400c8f0 comp rx=0 tx=0).stop 2024-09-13T16:43:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.547+0000 7fd669a00640 1 -- 172.21.15.28:0/1572011704 shutdown_connections 2024-09-13T16:43:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.547+0000 7fd669a00640 1 --2- 172.21.15.28:0/1572011704 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd64006aba0 0x7fd64006d060 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.547+0000 7fd669a00640 1 --2- 172.21.15.28:0/1572011704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd6640c4350 0x7fd66414db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.547+0000 7fd669a00640 1 --2- 172.21.15.28:0/1572011704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd6640c1830 0x7fd66414d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.547+0000 7fd669a00640 1 --2- 172.21.15.28:0/1572011704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6640706b0 0x7fd66415af50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:54.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.547+0000 7fd669a00640 1 -- 172.21.15.28:0/1572011704 >> 172.21.15.28:0/1572011704 conn(0x7fd6640c7390 msgr2=0x7fd6640c9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.547+0000 7fd669a00640 1 -- 172.21.15.28:0/1572011704 shutdown_connections 2024-09-13T16:43:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.548+0000 7fd669a00640 1 -- 172.21.15.28:0/1572011704 wait complete. 2024-09-13T16:43:54.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T16:43:54.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.739+0000 7fe430000640 1 Processor -- start 2024-09-13T16:43:54.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.739+0000 7fe430000640 1 -- start start 2024-09-13T16:43:54.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.740+0000 7fe430000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe42815a060 0x7fe4280c5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:54.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.740+0000 7fe430000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe42815a440 0x7fe42806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:54.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.740+0000 7fe430000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe4280c62a0 0x7fe42806dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:54.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.740+0000 7fe430000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4280c1d80 con 0x7fe42815a060 2024-09-13T16:43:54.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.740+0000 7fe430000640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe42815a920 con 0x7fe4280c62a0 2024-09-13T16:43:54.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.740+0000 7fe430000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe42815ac00 con 0x7fe42815a440 2024-09-13T16:43:54.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.741+0000 7fe42d600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe42815a060 0x7fe4280c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:54.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.741+0000 7fe42e000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe4280c62a0 0x7fe42806dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:54.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.741+0000 7fe42cc00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe42815a440 0x7fe42806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:54.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.741+0000 7fe42d600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe42815a060 0x7fe4280c5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47418/0 (socket says 172.21.15.28:47418) 2024-09-13T16:43:54.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.741+0000 7fe42d600640 1 -- 172.21.15.28:0/2774943934 learned_addr learned my addr 172.21.15.28:0/2774943934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.741+0000 7fe42d600640 1 -- 172.21.15.28:0/2774943934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe42815a440 msgr2=0x7fe42806d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.741+0000 7fe42d600640 1 --2- 172.21.15.28:0/2774943934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe42815a440 0x7fe42806d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.741+0000 7fe42d600640 1 -- 172.21.15.28:0/2774943934 >> v2:172.21.15.191:3300/0 conn(0x7fe4280c62a0 msgr2=0x7fe42806dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.741+0000 7fe42d600640 1 --2- 172.21.15.28:0/2774943934 >> v2:172.21.15.191:3300/0 conn(0x7fe4280c62a0 0x7fe42806dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.741+0000 7fe42d600640 1 -- 172.21.15.28:0/2774943934 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4280c2d50 con 0x7fe42815a060 2024-09-13T16:43:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.742+0000 7fe42d600640 1 --2- 172.21.15.28:0/2774943934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42815a060 0x7fe4280c5500 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe4280c8060 tx=0x7fe41c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50cc722af9865ae8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.743+0000 7fe427e00640 1 -- 172.21.15.28:0/2774943934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c032440 con 0x7fe42815a060 2024-09-13T16:43:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.743+0000 7fe427e00640 1 -- 172.21.15.28:0/2774943934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c0325a0 con 0x7fe42815a060 2024-09-13T16:43:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.743+0000 7fe427e00640 1 -- 172.21.15.28:0/2774943934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c032930 con 0x7fe42815a060 2024-09-13T16:43:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.743+0000 7fe430000640 1 -- 172.21.15.28:0/2774943934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42815a060 msgr2=0x7fe4280c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.743+0000 7fe430000640 1 --2- 172.21.15.28:0/2774943934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42815a060 0x7fe4280c5500 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe4280c8060 tx=0x7fe41c02a1e0 comp rx=0 tx=0).stop 2024-09-13T16:43:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.743+0000 7fe430000640 1 -- 172.21.15.28:0/2774943934 shutdown_connections 2024-09-13T16:43:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.743+0000 7fe430000640 1 --2- 172.21.15.28:0/2774943934 >> v2:172.21.15.191:3300/0 conn(0x7fe4280c62a0 0x7fe42806dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.743+0000 7fe430000640 1 --2- 172.21.15.28:0/2774943934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe42815a440 0x7fe42806d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.743+0000 7fe430000640 1 --2- 172.21.15.28:0/2774943934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42815a060 0x7fe4280c5500 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.743+0000 7fe430000640 1 -- 172.21.15.28:0/2774943934 >> 172.21.15.28:0/2774943934 conn(0x7fe4280ca6b0 msgr2=0x7fe4280caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.743+0000 7fe430000640 1 -- 172.21.15.28:0/2774943934 shutdown_connections 2024-09-13T16:43:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.743+0000 7fe430000640 1 -- 172.21.15.28:0/2774943934 wait complete. 2024-09-13T16:43:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.745+0000 7fe430000640 1 Processor -- start 2024-09-13T16:43:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.745+0000 7fe430000640 1 -- start start 2024-09-13T16:43:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.746+0000 7fe430000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4280c62a0 0x7fe42814cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.746+0000 7fe430000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42815a060 0x7fe4281472b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.746+0000 7fe430000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe42815a440 0x7fe4281f1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.746+0000 7fe430000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4280c8e40 con 0x7fe42815a060 2024-09-13T16:43:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.746+0000 7fe430000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4280c8580 con 0x7fe4280c62a0 2024-09-13T16:43:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.746+0000 7fe430000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe42814c890 con 0x7fe42815a440 2024-09-13T16:43:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.746+0000 7fe42d600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4280c62a0 0x7fe42814cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.746+0000 7fe42e000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe42815a440 0x7fe4281f1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.746+0000 7fe42cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42815a060 0x7fe4281472b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.746+0000 7fe42d600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4280c62a0 0x7fe42814cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:53346/0 (socket says 172.21.15.28:53346) 2024-09-13T16:43:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.746+0000 7fe42d600640 1 -- 172.21.15.28:0/3794968255 learned_addr learned my addr 172.21.15.28:0/3794968255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.747+0000 7fe42e000640 1 -- 172.21.15.28:0/3794968255 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4280c62a0 msgr2=0x7fe42814cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.747+0000 7fe42e000640 1 --2- 172.21.15.28:0/3794968255 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4280c62a0 0x7fe42814cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.747+0000 7fe42e000640 1 -- 172.21.15.28:0/3794968255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42815a060 msgr2=0x7fe4281472b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.747+0000 7fe42e000640 1 --2- 172.21.15.28:0/3794968255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42815a060 0x7fe4281472b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.747+0000 7fe42e000640 1 -- 172.21.15.28:0/3794968255 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41c0094d0 con 0x7fe42815a440 2024-09-13T16:43:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.747+0000 7fe42e000640 1 --2- 172.21.15.28:0/3794968255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe42815a440 0x7fe4281f1b00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe42000eba0 tx=0x7fe42000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.747+0000 7fe425600640 1 -- 172.21.15.28:0/3794968255 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42000cd20 con 0x7fe42815a440 2024-09-13T16:43:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.747+0000 7fe425600640 1 -- 172.21.15.28:0/3794968255 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42000f040 con 0x7fe42815a440 2024-09-13T16:43:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.747+0000 7fe425600640 1 -- 172.21.15.28:0/3794968255 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4200146c0 con 0x7fe42815a440 2024-09-13T16:43:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.747+0000 7fe430000640 1 -- 172.21.15.28:0/3794968255 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4280c87f0 con 0x7fe42815a440 2024-09-13T16:43:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.747+0000 7fe430000640 1 -- 172.21.15.28:0/3794968255 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42814cb70 con 0x7fe42815a440 2024-09-13T16:43:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.748+0000 7fe430000640 1 -- 172.21.15.28:0/3794968255 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4280c8ff0 con 0x7fe42815a440 2024-09-13T16:43:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.749+0000 7fe425600640 1 -- 172.21.15.28:0/3794968255 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe420014820 con 0x7fe42815a440 2024-09-13T16:43:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.750+0000 7fe425600640 1 --2- 172.21.15.28:0/3794968255 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe40806ac00 0x7fe40806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.750+0000 7fe425600640 1 -- 172.21.15.28:0/3794968255 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fe420089ff0 con 0x7fe42815a440 2024-09-13T16:43:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.750+0000 7fe42d600640 1 --2- 172.21.15.28:0/3794968255 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe40806ac00 0x7fe40806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.751+0000 7fe42d600640 1 --2- 172.21.15.28:0/3794968255 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe40806ac00 0x7fe40806d0c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe41c035040 tx=0x7fe41c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:54.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.755+0000 7fe425600640 1 -- 172.21.15.28:0/3794968255 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe4200bd9c0 con 0x7fe42815a440 2024-09-13T16:43:54.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.967+0000 7fe430000640 1 -- 172.21.15.28:0/3794968255 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe4280c5500 con 0x7fe42815a440 2024-09-13T16:43:54.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.975+0000 7fe425600640 1 -- 172.21.15.28:0/3794968255 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fe420018070 con 0x7fe42815a440 2024-09-13T16:43:54.977 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 2 and data pool 3 2024-09-13T16:43:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.979+0000 7fe430000640 1 -- 172.21.15.28:0/3794968255 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe40806ac00 msgr2=0x7fe40806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.979+0000 7fe430000640 1 --2- 172.21.15.28:0/3794968255 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe40806ac00 0x7fe40806d0c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe41c035040 tx=0x7fe41c02e000 comp rx=0 tx=0).stop 2024-09-13T16:43:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.979+0000 7fe430000640 1 -- 172.21.15.28:0/3794968255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe42815a440 msgr2=0x7fe4281f1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.979+0000 7fe430000640 1 --2- 172.21.15.28:0/3794968255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe42815a440 0x7fe4281f1b00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe42000eba0 tx=0x7fe42000c490 comp rx=0 tx=0).stop 2024-09-13T16:43:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.979+0000 7fe430000640 1 -- 172.21.15.28:0/3794968255 shutdown_connections 2024-09-13T16:43:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.979+0000 7fe430000640 1 --2- 172.21.15.28:0/3794968255 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe40806ac00 0x7fe40806d0c0 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe41c035040 tx=0x7fe41c02e000 comp rx=0 tx=0).stop 2024-09-13T16:43:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.979+0000 7fe430000640 1 --2- 172.21.15.28:0/3794968255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe42815a440 0x7fe4281f1b00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.979+0000 7fe430000640 1 --2- 172.21.15.28:0/3794968255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42815a060 0x7fe4281472b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.979+0000 7fe430000640 1 --2- 172.21.15.28:0/3794968255 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe4280c62a0 0x7fe42814cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.979+0000 7fe430000640 1 -- 172.21.15.28:0/3794968255 >> 172.21.15.28:0/3794968255 conn(0x7fe4280ca6b0 msgr2=0x7fe428069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:54.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.982+0000 7fe430000640 1 -- 172.21.15.28:0/3794968255 shutdown_connections 2024-09-13T16:43:54.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:54.983+0000 7fe430000640 1 -- 172.21.15.28:0/3794968255 wait complete. 2024-09-13T16:43:55.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:43:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.161+0000 7f7e4ac00640 1 Processor -- start 2024-09-13T16:43:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.162+0000 7f7e4ac00640 1 -- start start 2024-09-13T16:43:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.163+0000 7f7e4ac00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e4414f790 0x7f7e440cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.163+0000 7f7e4ac00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7e440c42f0 0x7f7e44070640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.163+0000 7f7e4ac00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e4406cb60 con 0x7f7e4414fb70 2024-09-13T16:43:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.163+0000 7f7e4ac00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e440c1c80 con 0x7f7e4414f790 2024-09-13T16:43:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.163+0000 7f7e4ac00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e4406ce40 con 0x7f7e440c42f0 2024-09-13T16:43:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.163+0000 7f7e48e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7e4414fb70 0x7f7e440c3140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34370/0 (socket says 172.21.15.28:34370) 2024-09-13T16:43:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.163+0000 7f7e48e00640 1 -- 172.21.15.28:0/2949214211 learned_addr learned my addr 172.21.15.28:0/2949214211 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:43:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.163+0000 7f7e4a200640 1 --2- 172.21.15.28:0/2949214211 >> v2:172.21.15.191:3300/0 conn(0x7f7e440c42f0 0x7f7e44070640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.163+0000 7f7e49800640 1 --2- 172.21.15.28:0/2949214211 >> v2:172.21.15.28:3300/0 conn(0x7f7e4414f790 0x7f7e440cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.163+0000 7f7e43e00640 1 -- 172.21.15.28:0/2949214211 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567195468 0 0) 0x7f7e4406cb60 con 0x7f7e4414fb70 2024-09-13T16:43:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.164+0000 7f7e43e00640 1 -- 172.21.15.28:0/2949214211 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e2c003520 con 0x7f7e4414fb70 2024-09-13T16:43:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.164+0000 7f7e49800640 1 -- 172.21.15.28:0/2949214211 >> v2:172.21.15.191:3300/0 conn(0x7f7e440c42f0 msgr2=0x7f7e44070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.164+0000 7f7e49800640 1 --2- 172.21.15.28:0/2949214211 >> v2:172.21.15.191:3300/0 conn(0x7f7e440c42f0 0x7f7e44070640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.164+0000 7f7e49800640 1 -- 172.21.15.28:0/2949214211 >> v1:172.21.15.28:6789/0 conn(0x7f7e4414fb70 legacy=0x7f7e440c3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.164+0000 7f7e49800640 1 -- 172.21.15.28:0/2949214211 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e44156070 con 0x7f7e4414f790 2024-09-13T16:43:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.164+0000 7f7e4a200640 1 --2- 172.21.15.28:0/2949214211 >> v2:172.21.15.191:3300/0 conn(0x7f7e440c42f0 0x7f7e44070640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:55.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.165+0000 7f7e49800640 1 --2- 172.21.15.28:0/2949214211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e4414f790 0x7f7e440cabd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7e4406ae20 tx=0x7f7e3802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1efebe65d1119cee server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:55.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.165+0000 7f7e43e00640 1 -- 172.21.15.28:0/2949214211 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e38007740 con 0x7f7e4414f790 2024-09-13T16:43:55.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.165+0000 7f7e43e00640 1 -- 172.21.15.28:0/2949214211 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3802d050 con 0x7f7e4414f790 2024-09-13T16:43:55.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.165+0000 7f7e43e00640 1 -- 172.21.15.28:0/2949214211 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e38007ac0 con 0x7f7e4414f790 2024-09-13T16:43:55.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.165+0000 7f7e4ac00640 1 -- 172.21.15.28:0/2949214211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e4414f790 msgr2=0x7f7e440cabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:55.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.165+0000 7f7e4ac00640 1 --2- 172.21.15.28:0/2949214211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e4414f790 0x7f7e440cabd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7e4406ae20 tx=0x7f7e3802a710 comp rx=0 tx=0).stop 2024-09-13T16:43:55.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.166+0000 7f7e4ac00640 1 -- 172.21.15.28:0/2949214211 shutdown_connections 2024-09-13T16:43:55.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.166+0000 7f7e4ac00640 1 --2- 172.21.15.28:0/2949214211 >> v2:172.21.15.191:3300/0 conn(0x7f7e440c42f0 0x7f7e44070640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.166+0000 7f7e4ac00640 1 --2- 172.21.15.28:0/2949214211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e4414f790 0x7f7e440cabd0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.166+0000 7f7e4ac00640 1 -- 172.21.15.28:0/2949214211 >> 172.21.15.28:0/2949214211 conn(0x7f7e440ca0e0 msgr2=0x7f7e440ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:55.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.166+0000 7f7e4ac00640 1 -- 172.21.15.28:0/2949214211 shutdown_connections 2024-09-13T16:43:55.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.166+0000 7f7e4ac00640 1 -- 172.21.15.28:0/2949214211 wait complete. 2024-09-13T16:43:55.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.167+0000 7f7e4ac00640 1 Processor -- start 2024-09-13T16:43:55.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.167+0000 7f7e4ac00640 1 -- start start 2024-09-13T16:43:55.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.168+0000 7f7e4ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e440c42f0 0x7f7e441dfa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.168+0000 7f7e4ac00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e4414f790 0x7f7e4414b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.168+0000 7f7e4ac00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e440c61a0 0x7f7e441f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.168+0000 7f7e4ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e441dffc0 con 0x7f7e440c42f0 2024-09-13T16:43:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.168+0000 7f7e4ac00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e44153700 con 0x7f7e440c61a0 2024-09-13T16:43:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.168+0000 7f7e4ac00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e441539e0 con 0x7f7e4414f790 2024-09-13T16:43:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.168+0000 7f7e49800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e440c42f0 0x7f7e441dfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.169+0000 7f7e48e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e4414f790 0x7f7e4414b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.169+0000 7f7e4a200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e440c61a0 0x7f7e441f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.169+0000 7f7e49800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e440c42f0 0x7f7e441dfa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47442/0 (socket says 172.21.15.28:47442) 2024-09-13T16:43:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.169+0000 7f7e49800640 1 -- 172.21.15.28:0/1842561893 learned_addr learned my addr 172.21.15.28:0/1842561893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.169+0000 7f7e48e00640 1 -- 172.21.15.28:0/1842561893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e440c61a0 msgr2=0x7f7e441f1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.169+0000 7f7e48e00640 1 --2- 172.21.15.28:0/1842561893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e440c61a0 0x7f7e441f1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.169+0000 7f7e48e00640 1 -- 172.21.15.28:0/1842561893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e440c42f0 msgr2=0x7f7e441dfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.169+0000 7f7e48e00640 1 --2- 172.21.15.28:0/1842561893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e440c42f0 0x7f7e441dfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.169+0000 7f7e48e00640 1 -- 172.21.15.28:0/1842561893 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c009400 con 0x7f7e4414f790 2024-09-13T16:43:55.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.169+0000 7f7e4a200640 1 --2- 172.21.15.28:0/1842561893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e440c61a0 0x7f7e441f1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:55.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.169+0000 7f7e48e00640 1 --2- 172.21.15.28:0/1842561893 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e4414f790 0x7f7e4414b810 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000c3c0 tx=0x7f7e30001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:55.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.169+0000 7f7e41600640 1 -- 172.21.15.28:0/1842561893 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e30005ce0 con 0x7f7e4414f790 2024-09-13T16:43:55.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.169+0000 7f7e41600640 1 -- 172.21.15.28:0/1842561893 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e30005e40 con 0x7f7e4414f790 2024-09-13T16:43:55.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.169+0000 7f7e4ac00640 1 -- 172.21.15.28:0/1842561893 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e38009a70 con 0x7f7e4414f790 2024-09-13T16:43:55.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.169+0000 7f7e41600640 1 -- 172.21.15.28:0/1842561893 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e300072d0 con 0x7f7e4414f790 2024-09-13T16:43:55.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.170+0000 7f7e4ac00640 1 -- 172.21.15.28:0/1842561893 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e44148210 con 0x7f7e4414f790 2024-09-13T16:43:55.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.171+0000 7f7e4ac00640 1 -- 172.21.15.28:0/1842561893 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e44071120 con 0x7f7e4414f790 2024-09-13T16:43:55.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.171+0000 7f7e41600640 1 -- 172.21.15.28:0/1842561893 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7e300074f0 con 0x7f7e4414f790 2024-09-13T16:43:55.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.172+0000 7f7e41600640 1 --2- 172.21.15.28:0/1842561893 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7e2406ac00 0x7f7e2406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.172+0000 7f7e41600640 1 -- 172.21.15.28:0/1842561893 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e30088920 con 0x7f7e4414f790 2024-09-13T16:43:55.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.173+0000 7f7e49800640 1 --2- 172.21.15.28:0/1842561893 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7e2406ac00 0x7f7e2406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.173+0000 7f7e49800640 1 --2- 172.21.15.28:0/1842561893 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7e2406ac00 0x7f7e2406d0c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7e38000f80 tx=0x7f7e3802f000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.177+0000 7f7e41600640 1 -- 172.21.15.28:0/1842561893 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7e30053a80 con 0x7f7e4414f790 2024-09-13T16:43:55.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.356+0000 7f7e4ac00640 1 -- 172.21.15.28:0/1842561893 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e4406fcd0 con 0x7f7e4414f790 2024-09-13T16:43:55.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.357+0000 7f7e41600640 1 -- 172.21.15.28:0/1842561893 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16494 (secure 0 0 0) 0x7f7e30017020 con 0x7f7e4414f790 2024-09-13T16:43:55.359 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:55.359 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":19,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:43:54.969374+0000","last_up_change":"2024-09-13T16:43:49.476313+0000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:43:49.643606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:43:52.828510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:43:53.945874+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6801","nonce":2964323435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6803","nonce":2964323435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6807","nonce":2964323435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6805","nonce":2964323435}]},"public_addr":"172.21.15.28:6801/2964323435","cluster_addr":"172.21.15.28:6803/2964323435","heartbeat_back_addr":"172.21.15.28:6807/2964323435","heartbeat_front_addr":"172.21.15.28:6805/2964323435","state":["exists","up"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6809","nonce":23768570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6811","nonce":23768570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6815","nonce":23768570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6813","nonce":23768570}]},"public_addr":"172.21.15.28:6809/23768570","cluster_addr":"172.21.15.28:6811/23768570","heartbeat_back_addr":"172.21.15.28:6815/23768570","heartbeat_front_addr":"172.21.15.28:6813/23768570","state":["exists","up"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6817","nonce":886887303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6819","nonce":886887303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6823","nonce":886887303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6821","nonce":886887303}]},"public_addr":"172.21.15.28:6817/886887303","cluster_addr":"172.21.15.28:6819/886887303","heartbeat_back_addr":"172.21.15.28:6823/886887303","heartbeat_front_addr":"172.21.15.28:6821/886887303","state":["exists","up"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6825","nonce":2616036674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6827","nonce":2616036674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6831","nonce":2616036674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6829","nonce":2616036674}]},"public_addr":"172.21.15.28:6825/2616036674","cluster_addr":"172.21.15.28:6827/2616036674","heartbeat_back_addr":"172.21.15.28:6831/2616036674","heartbeat_front_addr":"172.21.15.28:6829/2616036674","state":["exists","up"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6801","nonce":2817102450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6803","nonce":2817102450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6807","nonce":2817102450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6805","nonce":2817102450}]},"public_addr":"172.21.15.191:6801/2817102450","cluster_addr":"172.21.15.191:6803/2817102450","heartbeat_back_addr":"172.21.15.191:6807/2817102450","heartbeat_front_addr":"172.21.15.191:6805/2817102450","state":["exists","up"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6817","nonce":2503027971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6819","nonce":2503027971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6823","nonce":2503027971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6821","nonce":2503027971}]},"public_addr":"172.21.15.191:6817/2503027971","cluster_addr":"172.21.15.191:6819/2503027971","heartbeat_back_addr":"172.21.15.191:6823/2503027971","heartbeat_front_addr":"172.21.15.191:6821/2503027971","state":["exists","up"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6809","nonce":3006595088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6811","nonce":3006595088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6815","nonce":3006595088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6813","nonce":3006595088}]},"public_addr":"172.21.15.191:6809/3006595088","cluster_addr":"172.21.15.191:6811/3006595088","heartbeat_back_addr":"172.21.15.191:6815/3006595088","heartbeat_front_addr":"172.21.15.191:6813/3006595088","state":["exists","up"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6825","nonce":1477327720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6827","nonce":1477327720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6831","nonce":1477327720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6829","nonce":1477327720}]},"public_addr":"172.21.15.191:6825/1477327720","cluster_addr":"172.21.15.191:6827/1477327720","heartbeat_back_addr":"172.21.15.191:6831/1477327720","heartbeat_front_addr":"172.21.15.191:6829/1477327720","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.917250+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.954351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.017329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.046221+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.707467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.726947+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.782321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.764850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:43:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.359+0000 7f7e4ac00640 1 -- 172.21.15.28:0/1842561893 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7e2406ac00 msgr2=0x7f7e2406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.359+0000 7f7e4ac00640 1 --2- 172.21.15.28:0/1842561893 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7e2406ac00 0x7f7e2406d0c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7e38000f80 tx=0x7f7e3802f000 comp rx=0 tx=0).stop 2024-09-13T16:43:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.359+0000 7f7e4ac00640 1 -- 172.21.15.28:0/1842561893 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e4414f790 msgr2=0x7f7e4414b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.359+0000 7f7e4ac00640 1 --2- 172.21.15.28:0/1842561893 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e4414f790 0x7f7e4414b810 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000c3c0 tx=0x7f7e30001e70 comp rx=0 tx=0).stop 2024-09-13T16:43:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.359+0000 7f7e4ac00640 1 -- 172.21.15.28:0/1842561893 shutdown_connections 2024-09-13T16:43:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.359+0000 7f7e4ac00640 1 --2- 172.21.15.28:0/1842561893 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7e2406ac00 0x7f7e2406d0c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.359+0000 7f7e4ac00640 1 --2- 172.21.15.28:0/1842561893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e440c61a0 0x7f7e441f1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.359+0000 7f7e4ac00640 1 --2- 172.21.15.28:0/1842561893 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e4414f790 0x7f7e4414b810 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.359+0000 7f7e4ac00640 1 --2- 172.21.15.28:0/1842561893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e440c42f0 0x7f7e441dfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.359+0000 7f7e4ac00640 1 -- 172.21.15.28:0/1842561893 >> 172.21.15.28:0/1842561893 conn(0x7f7e440ca0e0 msgr2=0x7f7e44070500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:55.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.360+0000 7f7e4ac00640 1 -- 172.21.15.28:0/1842561893 shutdown_connections 2024-09-13T16:43:55.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.360+0000 7f7e4ac00640 1 -- 172.21.15.28:0/1842561893 wait complete. 2024-09-13T16:43:55.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:43:55.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.530+0000 7f8896400640 1 Processor -- start 2024-09-13T16:43:55.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.530+0000 7f8896400640 1 -- start start 2024-09-13T16:43:55.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.531+0000 7f8896400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f88900ca940 0x7f88900ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.531+0000 7f8896400640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f88900ce7f0 0x7f88900c9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.531+0000 7f8896400640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88900cbbe0 con 0x7f88900c46f0 2024-09-13T16:43:55.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.531+0000 7f8896400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88901532e0 con 0x7f88900ca940 2024-09-13T16:43:55.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.531+0000 7f8896400640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88900c94f0 con 0x7f88900ce7f0 2024-09-13T16:43:55.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.531+0000 7f8895a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f88900ce7f0 0x7f88900c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.531+0000 7f8895000640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f88900c46f0 0x7f8890159500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:46992/0 (socket says 172.21.15.28:46992) 2024-09-13T16:43:55.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.531+0000 7f8895000640 1 -- 172.21.15.28:0/4054483129 learned_addr learned my addr 172.21.15.28:0/4054483129 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:43:55.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.531+0000 7f888fe00640 1 --2- 172.21.15.28:0/4054483129 >> v2:172.21.15.28:3300/0 conn(0x7f88900ca940 0x7f88900ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.532+0000 7f888fe00640 1 -- 172.21.15.28:0/4054483129 >> v2:172.21.15.191:3300/0 conn(0x7f88900ce7f0 msgr2=0x7f88900c9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:55.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.532+0000 7f888fe00640 1 --2- 172.21.15.28:0/4054483129 >> v2:172.21.15.191:3300/0 conn(0x7f88900ce7f0 0x7f88900c9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.532+0000 7f888fe00640 1 -- 172.21.15.28:0/4054483129 >> v1:172.21.15.191:6789/0 conn(0x7f88900c46f0 legacy=0x7f8890159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:55.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.532+0000 7f888fe00640 1 -- 172.21.15.28:0/4054483129 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88900cb060 con 0x7f88900ca940 2024-09-13T16:43:55.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.533+0000 7f888fe00640 1 --2- 172.21.15.28:0/4054483129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88900ca940 0x7f88900ce2b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f88900b3cb0 tx=0x7f887c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b402419ab03c0da4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:55.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.533+0000 7f888f400640 1 -- 172.21.15.28:0/4054483129 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c02ae60 con 0x7f88900ca940 2024-09-13T16:43:55.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.533+0000 7f888f400640 1 -- 172.21.15.28:0/4054483129 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887c032440 con 0x7f88900ca940 2024-09-13T16:43:55.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.533+0000 7f888f400640 1 -- 172.21.15.28:0/4054483129 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c0327d0 con 0x7f88900ca940 2024-09-13T16:43:55.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.534+0000 7f8896400640 1 -- 172.21.15.28:0/4054483129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88900ca940 msgr2=0x7f88900ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:55.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.534+0000 7f8896400640 1 --2- 172.21.15.28:0/4054483129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88900ca940 0x7f88900ce2b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f88900b3cb0 tx=0x7f887c02a110 comp rx=0 tx=0).stop 2024-09-13T16:43:55.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.534+0000 7f8896400640 1 -- 172.21.15.28:0/4054483129 shutdown_connections 2024-09-13T16:43:55.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.534+0000 7f8896400640 1 --2- 172.21.15.28:0/4054483129 >> v2:172.21.15.191:3300/0 conn(0x7f88900ce7f0 0x7f88900c9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.534+0000 7f8896400640 1 --2- 172.21.15.28:0/4054483129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88900ca940 0x7f88900ce2b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.534+0000 7f8896400640 1 -- 172.21.15.28:0/4054483129 >> 172.21.15.28:0/4054483129 conn(0x7f889006c830 msgr2=0x7f889006a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:55.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.534+0000 7f8896400640 1 -- 172.21.15.28:0/4054483129 shutdown_connections 2024-09-13T16:43:55.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.534+0000 7f8896400640 1 -- 172.21.15.28:0/4054483129 wait complete. 2024-09-13T16:43:55.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.536+0000 7f8896400640 1 Processor -- start 2024-09-13T16:43:55.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.536+0000 7f8896400640 1 -- start start 2024-09-13T16:43:55.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.536+0000 7f8896400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88900c46f0 0x7f889014ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.536+0000 7f8896400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88900ca940 0x7f8890157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.537+0000 7f8896400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88900ce7f0 0x7f8890157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.537+0000 7f8896400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88901f1d90 con 0x7f88900ce7f0 2024-09-13T16:43:55.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.537+0000 7f8896400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8890154e50 con 0x7f88900ca940 2024-09-13T16:43:55.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.537+0000 7f8896400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88901570b0 con 0x7f88900c46f0 2024-09-13T16:43:55.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.537+0000 7f8895000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88900c46f0 0x7f889014ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.537+0000 7f8895a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88900ce7f0 0x7f8890157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.537+0000 7f888fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88900ca940 0x7f8890157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.537+0000 7f8895000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88900c46f0 0x7f889014ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:57594/0 (socket says 172.21.15.28:57594) 2024-09-13T16:43:55.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.537+0000 7f888fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88900ca940 0x7f8890157560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:53386/0 (socket says 172.21.15.28:53386) 2024-09-13T16:43:55.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.537+0000 7f8895000640 1 -- 172.21.15.28:0/1097709521 learned_addr learned my addr 172.21.15.28:0/1097709521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:55.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.537+0000 7f8895000640 1 -- 172.21.15.28:0/1097709521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88900ca940 msgr2=0x7f8890157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:55.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.537+0000 7f8895000640 1 --2- 172.21.15.28:0/1097709521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88900ca940 0x7f8890157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.537+0000 7f8895000640 1 -- 172.21.15.28:0/1097709521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88900ce7f0 msgr2=0x7f8890157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:55.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.537+0000 7f8895000640 1 --2- 172.21.15.28:0/1097709521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88900ce7f0 0x7f8890157aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.537+0000 7f8895000640 1 -- 172.21.15.28:0/1097709521 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c009400 con 0x7f88900c46f0 2024-09-13T16:43:55.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.537+0000 7f888fe00640 1 --2- 172.21.15.28:0/1097709521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88900ca940 0x7f8890157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:55.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.537+0000 7f8895000640 1 --2- 172.21.15.28:0/1097709521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88900c46f0 0x7f889014ec30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f888400c480 tx=0x7f888400c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:55.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.538+0000 7f888cc00640 1 -- 172.21.15.28:0/1097709521 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884005ce0 con 0x7f88900c46f0 2024-09-13T16:43:55.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.538+0000 7f888cc00640 1 -- 172.21.15.28:0/1097709521 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8884005e40 con 0x7f88900c46f0 2024-09-13T16:43:55.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.538+0000 7f888cc00640 1 -- 172.21.15.28:0/1097709521 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88840072f0 con 0x7f88900c46f0 2024-09-13T16:43:55.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.538+0000 7f8896400640 1 -- 172.21.15.28:0/1097709521 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8890156d90 con 0x7f88900c46f0 2024-09-13T16:43:55.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.538+0000 7f8896400640 1 -- 172.21.15.28:0/1097709521 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88901550a0 con 0x7f88900c46f0 2024-09-13T16:43:55.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.539+0000 7f8896400640 1 -- 172.21.15.28:0/1097709521 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8890067460 con 0x7f88900c46f0 2024-09-13T16:43:55.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.540+0000 7f888cc00640 1 -- 172.21.15.28:0/1097709521 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8884007450 con 0x7f88900c46f0 2024-09-13T16:43:55.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.540+0000 7f888cc00640 1 --2- 172.21.15.28:0/1097709521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f887006ac30 0x7f887006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.540+0000 7f888cc00640 1 -- 172.21.15.28:0/1097709521 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8884088810 con 0x7f88900c46f0 2024-09-13T16:43:55.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.541+0000 7f888fe00640 1 --2- 172.21.15.28:0/1097709521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f887006ac30 0x7f887006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.541+0000 7f888fe00640 1 --2- 172.21.15.28:0/1097709521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f887006ac30 0x7f887006d0f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f887c034000 tx=0x7f887c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:55.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.545+0000 7f888cc00640 1 -- 172.21.15.28:0/1097709521 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8884053a90 con 0x7f88900c46f0 2024-09-13T16:43:55.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.726+0000 7f8896400640 1 -- 172.21.15.28:0/1097709521 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88900b4110 con 0x7f88900c46f0 2024-09-13T16:43:55.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.728+0000 7f888cc00640 1 -- 172.21.15.28:0/1097709521 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16494 (secure 0 0 0) 0x7f8884017090 con 0x7f88900c46f0 2024-09-13T16:43:55.730 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:43:55.730 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":19,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:43:54.969374+0000","last_up_change":"2024-09-13T16:43:49.476313+0000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:43:49.643606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:43:52.828510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:43:53.945874+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6801","nonce":2964323435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6803","nonce":2964323435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6807","nonce":2964323435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6805","nonce":2964323435}]},"public_addr":"172.21.15.28:6801/2964323435","cluster_addr":"172.21.15.28:6803/2964323435","heartbeat_back_addr":"172.21.15.28:6807/2964323435","heartbeat_front_addr":"172.21.15.28:6805/2964323435","state":["exists","up"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6809","nonce":23768570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6811","nonce":23768570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6815","nonce":23768570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6813","nonce":23768570}]},"public_addr":"172.21.15.28:6809/23768570","cluster_addr":"172.21.15.28:6811/23768570","heartbeat_back_addr":"172.21.15.28:6815/23768570","heartbeat_front_addr":"172.21.15.28:6813/23768570","state":["exists","up"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6817","nonce":886887303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6819","nonce":886887303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6823","nonce":886887303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6821","nonce":886887303}]},"public_addr":"172.21.15.28:6817/886887303","cluster_addr":"172.21.15.28:6819/886887303","heartbeat_back_addr":"172.21.15.28:6823/886887303","heartbeat_front_addr":"172.21.15.28:6821/886887303","state":["exists","up"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6825","nonce":2616036674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6827","nonce":2616036674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6831","nonce":2616036674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6829","nonce":2616036674}]},"public_addr":"172.21.15.28:6825/2616036674","cluster_addr":"172.21.15.28:6827/2616036674","heartbeat_back_addr":"172.21.15.28:6831/2616036674","heartbeat_front_addr":"172.21.15.28:6829/2616036674","state":["exists","up"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6801","nonce":2817102450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6803","nonce":2817102450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6807","nonce":2817102450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6805","nonce":2817102450}]},"public_addr":"172.21.15.191:6801/2817102450","cluster_addr":"172.21.15.191:6803/2817102450","heartbeat_back_addr":"172.21.15.191:6807/2817102450","heartbeat_front_addr":"172.21.15.191:6805/2817102450","state":["exists","up"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6817","nonce":2503027971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6819","nonce":2503027971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6823","nonce":2503027971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6821","nonce":2503027971}]},"public_addr":"172.21.15.191:6817/2503027971","cluster_addr":"172.21.15.191:6819/2503027971","heartbeat_back_addr":"172.21.15.191:6823/2503027971","heartbeat_front_addr":"172.21.15.191:6821/2503027971","state":["exists","up"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6809","nonce":3006595088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6811","nonce":3006595088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6815","nonce":3006595088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6813","nonce":3006595088}]},"public_addr":"172.21.15.191:6809/3006595088","cluster_addr":"172.21.15.191:6811/3006595088","heartbeat_back_addr":"172.21.15.191:6815/3006595088","heartbeat_front_addr":"172.21.15.191:6813/3006595088","state":["exists","up"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6825","nonce":1477327720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6827","nonce":1477327720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6831","nonce":1477327720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6829","nonce":1477327720}]},"public_addr":"172.21.15.191:6825/1477327720","cluster_addr":"172.21.15.191:6827/1477327720","heartbeat_back_addr":"172.21.15.191:6831/1477327720","heartbeat_front_addr":"172.21.15.191:6829/1477327720","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.917250+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.954351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.017329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.046221+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.707467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.726947+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.782321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.764850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:43:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.730+0000 7f8896400640 1 -- 172.21.15.28:0/1097709521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f887006ac30 msgr2=0x7f887006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.730+0000 7f8896400640 1 --2- 172.21.15.28:0/1097709521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f887006ac30 0x7f887006d0f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f887c034000 tx=0x7f887c02e000 comp rx=0 tx=0).stop 2024-09-13T16:43:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.730+0000 7f8896400640 1 -- 172.21.15.28:0/1097709521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88900c46f0 msgr2=0x7f889014ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.730+0000 7f8896400640 1 --2- 172.21.15.28:0/1097709521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88900c46f0 0x7f889014ec30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f888400c480 tx=0x7f888400c940 comp rx=0 tx=0).stop 2024-09-13T16:43:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.730+0000 7f8896400640 1 -- 172.21.15.28:0/1097709521 shutdown_connections 2024-09-13T16:43:55.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.730+0000 7f8896400640 1 --2- 172.21.15.28:0/1097709521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f887006ac30 0x7f887006d0f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.730+0000 7f8896400640 1 --2- 172.21.15.28:0/1097709521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88900ce7f0 0x7f8890157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.730+0000 7f8896400640 1 --2- 172.21.15.28:0/1097709521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f88900ca940 0x7f8890157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.730+0000 7f8896400640 1 --2- 172.21.15.28:0/1097709521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f88900c46f0 0x7f889014ec30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.730+0000 7f8896400640 1 -- 172.21.15.28:0/1097709521 >> 172.21.15.28:0/1097709521 conn(0x7f889006c830 msgr2=0x7f88900cd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:55.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.730+0000 7f8896400640 1 -- 172.21.15.28:0/1097709521 shutdown_connections 2024-09-13T16:43:55.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.730+0000 7f8896400640 1 -- 172.21.15.28:0/1097709521 wait complete. 2024-09-13T16:43:55.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T16:43:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.904+0000 7f6310a00640 1 Processor -- start 2024-09-13T16:43:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.904+0000 7f6310a00640 1 -- start start 2024-09-13T16:43:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.905+0000 7f6310a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f630c06af90 0x7f630c0c1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.905+0000 7f6310a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f630c06c340 0x7f630c0c2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.905+0000 7f6310a00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f630c0c2980 con 0x7f630c1563f0 2024-09-13T16:43:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.905+0000 7f6310a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630c0c7fd0 con 0x7f630c06c340 2024-09-13T16:43:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.905+0000 7f6310a00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630c0c2c00 con 0x7f630c06af90 2024-09-13T16:43:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.905+0000 7f6309400640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f630c06af90 0x7f630c0c1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.905+0000 7f630a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f630c06c340 0x7f630c0c2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.905+0000 7f6309e00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f630c1563f0 0x7f630c1591f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:47000/0 (socket says 172.21.15.28:47000) 2024-09-13T16:43:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.905+0000 7f6309400640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f630c06af90 0x7f630c0c1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:53390/0 (socket says 172.21.15.28:53390) 2024-09-13T16:43:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.905+0000 7f6309e00640 1 -- 172.21.15.28:0/3276696582 learned_addr learned my addr 172.21.15.28:0/3276696582 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:43:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.906+0000 7f6309400640 1 -- 172.21.15.28:0/3276696582 >> v2:172.21.15.28:3300/0 conn(0x7f630c06c340 msgr2=0x7f630c0c2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.906+0000 7f6309400640 1 --2- 172.21.15.28:0/3276696582 >> v2:172.21.15.28:3300/0 conn(0x7f630c06c340 0x7f630c0c2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.906+0000 7f6309400640 1 -- 172.21.15.28:0/3276696582 >> v1:172.21.15.191:6789/0 conn(0x7f630c1563f0 legacy=0x7f630c1591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.906+0000 7f6309400640 1 -- 172.21.15.28:0/3276696582 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630c158cd0 con 0x7f630c06af90 2024-09-13T16:43:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.906+0000 7f6308a00640 1 -- 172.21.15.28:0/3276696582 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603179756 0 0) 0x7f630c0c2980 con 0x7f630c1563f0 2024-09-13T16:43:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.907+0000 7f6309400640 1 --2- 172.21.15.28:0/3276696582 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f630c06af90 0x7f630c0c1f00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f630c06d170 tx=0x7f62f802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebe18af71952547d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.907+0000 7f6308a00640 1 -- 172.21.15.28:0/3276696582 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f8032440 con 0x7f630c06af90 2024-09-13T16:43:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.907+0000 7f6308a00640 1 -- 172.21.15.28:0/3276696582 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f80325a0 con 0x7f630c06af90 2024-09-13T16:43:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.907+0000 7f6308a00640 1 -- 172.21.15.28:0/3276696582 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f8032930 con 0x7f630c06af90 2024-09-13T16:43:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.908+0000 7f6310a00640 1 -- 172.21.15.28:0/3276696582 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f630c06af90 msgr2=0x7f630c0c1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.908+0000 7f6310a00640 1 --2- 172.21.15.28:0/3276696582 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f630c06af90 0x7f630c0c1f00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f630c06d170 tx=0x7f62f802a1a0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.908+0000 7f6310a00640 1 -- 172.21.15.28:0/3276696582 shutdown_connections 2024-09-13T16:43:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.908+0000 7f6310a00640 1 --2- 172.21.15.28:0/3276696582 >> v2:172.21.15.28:3300/0 conn(0x7f630c06c340 0x7f630c0c2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.908+0000 7f6310a00640 1 --2- 172.21.15.28:0/3276696582 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f630c06af90 0x7f630c0c1f00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.908+0000 7f6310a00640 1 -- 172.21.15.28:0/3276696582 >> 172.21.15.28:0/3276696582 conn(0x7f630c06fc30 msgr2=0x7f630c070010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.908+0000 7f6310a00640 1 -- 172.21.15.28:0/3276696582 shutdown_connections 2024-09-13T16:43:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.908+0000 7f6310a00640 1 -- 172.21.15.28:0/3276696582 wait complete. 2024-09-13T16:43:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.910+0000 7f6310a00640 1 Processor -- start 2024-09-13T16:43:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.910+0000 7f6310a00640 1 -- start start 2024-09-13T16:43:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.911+0000 7f6310a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f630c06af90 0x7f630c1e42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.911+0000 7f6310a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f630c1563f0 0x7f630c1f1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.911+0000 7f6310a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f630c1518e0 0x7f630c1f2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.911+0000 7f6310a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630c1e0a70 con 0x7f630c1518e0 2024-09-13T16:43:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.911+0000 7f6310a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630c150f80 con 0x7f630c1563f0 2024-09-13T16:43:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.911+0000 7f6310a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630c151540 con 0x7f630c06af90 2024-09-13T16:43:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.911+0000 7f630a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f630c1518e0 0x7f630c1f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.911+0000 7f6309e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f630c06af90 0x7f630c1e42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.911+0000 7f6309400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f630c1563f0 0x7f630c1f1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.911+0000 7f6309400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f630c1563f0 0x7f630c1f1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:53402/0 (socket says 172.21.15.28:53402) 2024-09-13T16:43:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.911+0000 7f6309e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f630c06af90 0x7f630c1e42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:57608/0 (socket says 172.21.15.28:57608) 2024-09-13T16:43:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.911+0000 7f6309e00640 1 -- 172.21.15.28:0/618983939 learned_addr learned my addr 172.21.15.28:0/618983939 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.911+0000 7f630a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f630c1518e0 0x7f630c1f2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47484/0 (socket says 172.21.15.28:47484) 2024-09-13T16:43:55.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.911+0000 7f630a800640 1 -- 172.21.15.28:0/618983939 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f630c06af90 msgr2=0x7f630c1e42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:55.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.911+0000 7f630a800640 1 --2- 172.21.15.28:0/618983939 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f630c06af90 0x7f630c1e42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.911+0000 7f630a800640 1 -- 172.21.15.28:0/618983939 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f630c1563f0 msgr2=0x7f630c1f1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:55.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.912+0000 7f630a800640 1 --2- 172.21.15.28:0/618983939 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f630c1563f0 0x7f630c1f1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:55.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.912+0000 7f630a800640 1 -- 172.21.15.28:0/618983939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f8009400 con 0x7f630c1518e0 2024-09-13T16:43:55.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.912+0000 7f6309400640 1 --2- 172.21.15.28:0/618983939 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f630c1563f0 0x7f630c1f1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:43:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.912+0000 7f630a800640 1 --2- 172.21.15.28:0/618983939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f630c1518e0 0x7f630c1f2090 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f630400efc0 tx=0x7f630400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.912+0000 7f6302000640 1 -- 172.21.15.28:0/618983939 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6304009c00 con 0x7f630c1518e0 2024-09-13T16:43:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.912+0000 7f6302000640 1 -- 172.21.15.28:0/618983939 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630400f040 con 0x7f630c1518e0 2024-09-13T16:43:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.912+0000 7f6302000640 1 -- 172.21.15.28:0/618983939 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63040146c0 con 0x7f630c1518e0 2024-09-13T16:43:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.912+0000 7f6310a00640 1 -- 172.21.15.28:0/618983939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f630c1f26f0 con 0x7f630c1518e0 2024-09-13T16:43:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.912+0000 7f6310a00640 1 -- 172.21.15.28:0/618983939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f630c150890 con 0x7f630c1518e0 2024-09-13T16:43:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.913+0000 7f6310a00640 1 -- 172.21.15.28:0/618983939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630c0c74a0 con 0x7f630c1518e0 2024-09-13T16:43:55.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.914+0000 7f6302000640 1 -- 172.21.15.28:0/618983939 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6304014820 con 0x7f630c1518e0 2024-09-13T16:43:55.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.915+0000 7f6302000640 1 --2- 172.21.15.28:0/618983939 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f62ec06ac30 0x7f62ec06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:55.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.915+0000 7f6302000640 1 -- 172.21.15.28:0/618983939 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6304089fe0 con 0x7f630c1518e0 2024-09-13T16:43:55.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.915+0000 7f6309e00640 1 --2- 172.21.15.28:0/618983939 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f62ec06ac30 0x7f62ec06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:55.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.916+0000 7f6309e00640 1 --2- 172.21.15.28:0/618983939 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f62ec06ac30 0x7f62ec06d0f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f62fc000f60 tx=0x7f62fc007ea0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:55.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:55.920+0000 7f6302000640 1 -- 172.21.15.28:0/618983939 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6304055260 con 0x7f630c1518e0 2024-09-13T16:43:56.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:56.137+0000 7f6310a00640 1 -- 172.21.15.28:0/618983939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f630c151680 con 0x7f630c1518e0 2024-09-13T16:43:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:56.976+0000 7f6302000640 1 -- 172.21.15.28:0/618983939 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7f63040180b0 con 0x7f630c1518e0 2024-09-13T16:43:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:56.979+0000 7f6310a00640 1 -- 172.21.15.28:0/618983939 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f62ec06ac30 msgr2=0x7f62ec06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:56.979+0000 7f6310a00640 1 --2- 172.21.15.28:0/618983939 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f62ec06ac30 0x7f62ec06d0f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f62fc000f60 tx=0x7f62fc007ea0 comp rx=0 tx=0).stop 2024-09-13T16:43:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:56.979+0000 7f6310a00640 1 -- 172.21.15.28:0/618983939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f630c1518e0 msgr2=0x7f630c1f2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:56.979+0000 7f6310a00640 1 --2- 172.21.15.28:0/618983939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f630c1518e0 0x7f630c1f2090 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f630400efc0 tx=0x7f630400c490 comp rx=0 tx=0).stop 2024-09-13T16:43:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:56.979+0000 7f6310a00640 1 -- 172.21.15.28:0/618983939 shutdown_connections 2024-09-13T16:43:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:56.979+0000 7f6310a00640 1 --2- 172.21.15.28:0/618983939 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f62ec06ac30 0x7f62ec06d0f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:56.979+0000 7f6310a00640 1 --2- 172.21.15.28:0/618983939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f630c1518e0 0x7f630c1f2090 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:56.979+0000 7f6310a00640 1 --2- 172.21.15.28:0/618983939 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f630c1563f0 0x7f630c1f1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:56.979+0000 7f6310a00640 1 --2- 172.21.15.28:0/618983939 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f630c06af90 0x7f630c1e42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:56.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:56.979+0000 7f6310a00640 1 -- 172.21.15.28:0/618983939 >> 172.21.15.28:0/618983939 conn(0x7f630c06fc30 msgr2=0x7f630c0c8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:56.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:56.979+0000 7f6310a00640 1 -- 172.21.15.28:0/618983939 shutdown_connections 2024-09-13T16:43:56.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:56.979+0000 7f6310a00640 1 -- 172.21.15.28:0/618983939 wait complete. 2024-09-13T16:43:56.998 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300} 2024-09-13T16:43:56.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-09-13T16:43:57.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.159+0000 7ff611200640 1 Processor -- start 2024-09-13T16:43:57.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.159+0000 7ff611200640 1 -- start start 2024-09-13T16:43:57.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.160+0000 7ff611200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff60c0c7f10 0x7ff60c0c6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:57.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.160+0000 7ff611200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff60c0c5c00 0x7ff60c158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.160+0000 7ff611200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff60c0c82f0 0x7ff60c1590f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.160+0000 7ff611200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60c159630 con 0x7ff60c0c82f0 2024-09-13T16:43:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.160+0000 7ff611200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60c06f760 con 0x7ff60c0c7f10 2024-09-13T16:43:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.160+0000 7ff611200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0c6e70 con 0x7ff60c0c5c00 2024-09-13T16:43:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.161+0000 7ff60be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff60c0c82f0 0x7ff60c1590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.161+0000 7ff60be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff60c0c82f0 0x7ff60c1590f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47490/0 (socket says 172.21.15.28:47490) 2024-09-13T16:43:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.161+0000 7ff60be00640 1 -- 172.21.15.28:0/1374590672 learned_addr learned my addr 172.21.15.28:0/1374590672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.161+0000 7ff603e00640 1 --2- 172.21.15.28:0/1374590672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff60c0c5c00 0x7ff60c158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.161+0000 7ff60b400640 1 --2- 172.21.15.28:0/1374590672 >> v2:172.21.15.191:3300/0 conn(0x7ff60c0c7f10 0x7ff60c0c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.161+0000 7ff60be00640 1 -- 172.21.15.28:0/1374590672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff60c0c5c00 msgr2=0x7ff60c158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.161+0000 7ff60be00640 1 --2- 172.21.15.28:0/1374590672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff60c0c5c00 0x7ff60c158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.161+0000 7ff60be00640 1 -- 172.21.15.28:0/1374590672 >> v2:172.21.15.191:3300/0 conn(0x7ff60c0c7f10 msgr2=0x7ff60c0c6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.161+0000 7ff60be00640 1 --2- 172.21.15.28:0/1374590672 >> v2:172.21.15.191:3300/0 conn(0x7ff60c0c7f10 0x7ff60c0c6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.161+0000 7ff60be00640 1 -- 172.21.15.28:0/1374590672 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c0c7110 con 0x7ff60c0c82f0 2024-09-13T16:43:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.162+0000 7ff60be00640 1 --2- 172.21.15.28:0/1374590672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c0c82f0 0x7ff60c1590f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0cc3c0 tx=0x7ff60402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=226e908287e933d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.162+0000 7ff60aa00640 1 -- 172.21.15.28:0/1374590672 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604031600 con 0x7ff60c0c82f0 2024-09-13T16:43:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.162+0000 7ff60aa00640 1 -- 172.21.15.28:0/1374590672 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60402d050 con 0x7ff60c0c82f0 2024-09-13T16:43:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.162+0000 7ff60aa00640 1 -- 172.21.15.28:0/1374590672 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604031980 con 0x7ff60c0c82f0 2024-09-13T16:43:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.163+0000 7ff611200640 1 -- 172.21.15.28:0/1374590672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c0c82f0 msgr2=0x7ff60c1590f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.163+0000 7ff611200640 1 --2- 172.21.15.28:0/1374590672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c0c82f0 0x7ff60c1590f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0cc3c0 tx=0x7ff60402a730 comp rx=0 tx=0).stop 2024-09-13T16:43:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.163+0000 7ff611200640 1 -- 172.21.15.28:0/1374590672 shutdown_connections 2024-09-13T16:43:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.163+0000 7ff611200640 1 --2- 172.21.15.28:0/1374590672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c0c82f0 0x7ff60c1590f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.163+0000 7ff611200640 1 --2- 172.21.15.28:0/1374590672 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff60c0c5c00 0x7ff60c158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.163+0000 7ff611200640 1 --2- 172.21.15.28:0/1374590672 >> v2:172.21.15.191:3300/0 conn(0x7ff60c0c7f10 0x7ff60c0c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.163+0000 7ff611200640 1 -- 172.21.15.28:0/1374590672 >> 172.21.15.28:0/1374590672 conn(0x7ff60c06af90 msgr2=0x7ff60c06b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.163+0000 7ff611200640 1 -- 172.21.15.28:0/1374590672 shutdown_connections 2024-09-13T16:43:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.163+0000 7ff611200640 1 -- 172.21.15.28:0/1374590672 wait complete. 2024-09-13T16:43:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.165+0000 7ff611200640 1 Processor -- start 2024-09-13T16:43:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.165+0000 7ff611200640 1 -- start start 2024-09-13T16:43:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.165+0000 7ff611200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff60c0c5c00 0x7ff60c0c1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.166+0000 7ff611200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c0c7f10 0x7ff60c124ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.166+0000 7ff611200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff60c0c82f0 0x7ff60c125420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.166+0000 7ff611200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0be200 con 0x7ff60c0c7f10 2024-09-13T16:43:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.166+0000 7ff611200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0be4d0 con 0x7ff60c0c5c00 2024-09-13T16:43:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.166+0000 7ff611200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0bea70 con 0x7ff60c0c82f0 2024-09-13T16:43:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.166+0000 7ff603e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c0c7f10 0x7ff60c124ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.166+0000 7ff603e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c0c7f10 0x7ff60c124ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47500/0 (socket says 172.21.15.28:47500) 2024-09-13T16:43:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.166+0000 7ff60b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff60c0c5c00 0x7ff60c0c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.166+0000 7ff603e00640 1 -- 172.21.15.28:0/2247973040 learned_addr learned my addr 172.21.15.28:0/2247973040 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.166+0000 7ff60be00640 1 --2- 172.21.15.28:0/2247973040 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff60c0c82f0 0x7ff60c125420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.166+0000 7ff603e00640 1 -- 172.21.15.28:0/2247973040 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff60c0c82f0 msgr2=0x7ff60c125420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.166+0000 7ff603e00640 1 --2- 172.21.15.28:0/2247973040 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff60c0c82f0 0x7ff60c125420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.166+0000 7ff603e00640 1 -- 172.21.15.28:0/2247973040 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff60c0c5c00 msgr2=0x7ff60c0c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.166+0000 7ff603e00640 1 --2- 172.21.15.28:0/2247973040 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff60c0c5c00 0x7ff60c0c1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.166+0000 7ff603e00640 1 -- 172.21.15.28:0/2247973040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff604009b30 con 0x7ff60c0c7f10 2024-09-13T16:43:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.166+0000 7ff603e00640 1 --2- 172.21.15.28:0/2247973040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c0c7f10 0x7ff60c124ee0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400e890 tx=0x7ff5f400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.167+0000 7ff603400640 1 -- 172.21.15.28:0/2247973040 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f400c8e0 con 0x7ff60c0c7f10 2024-09-13T16:43:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.167+0000 7ff603400640 1 -- 172.21.15.28:0/2247973040 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f400ca40 con 0x7ff60c0c7f10 2024-09-13T16:43:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.167+0000 7ff603400640 1 -- 172.21.15.28:0/2247973040 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f400cdd0 con 0x7ff60c0c7f10 2024-09-13T16:43:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.167+0000 7ff611200640 1 -- 172.21.15.28:0/2247973040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60c0be790 con 0x7ff60c0c7f10 2024-09-13T16:43:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.167+0000 7ff611200640 1 -- 172.21.15.28:0/2247973040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60c0cadd0 con 0x7ff60c0c7f10 2024-09-13T16:43:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.168+0000 7ff611200640 1 -- 172.21.15.28:0/2247973040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff60c15cca0 con 0x7ff60c0c7f10 2024-09-13T16:43:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.169+0000 7ff603400640 1 -- 172.21.15.28:0/2247973040 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff5f4013470 con 0x7ff60c0c7f10 2024-09-13T16:43:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.170+0000 7ff603400640 1 --2- 172.21.15.28:0/2247973040 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff5e806ac30 0x7ff5e806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.170+0000 7ff603400640 1 -- 172.21.15.28:0/2247973040 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5f4089420 con 0x7ff60c0c7f10 2024-09-13T16:43:57.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.170+0000 7ff60b400640 1 --2- 172.21.15.28:0/2247973040 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff5e806ac30 0x7ff5e806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:57.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.171+0000 7ff60b400640 1 --2- 172.21.15.28:0/2247973040 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff5e806ac30 0x7ff5e806d0f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc009b40 tx=0x7ff5fc009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.174+0000 7ff603400640 1 -- 172.21.15.28:0/2247973040 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff5f40546a0 con 0x7ff60c0c7f10 2024-09-13T16:43:57.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.390+0000 7ff611200640 1 -- 172.21.15.28:0/2247973040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7ff60c0c9f90 con 0x7ff60c0c7f10 2024-09-13T16:43:57.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.976+0000 7ff603400640 1 -- 172.21.15.28:0/2247973040 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7ff5f4057cc0 con 0x7ff60c0c7f10 2024-09-13T16:43:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.979+0000 7ff611200640 1 -- 172.21.15.28:0/2247973040 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff5e806ac30 msgr2=0x7ff5e806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.979+0000 7ff611200640 1 --2- 172.21.15.28:0/2247973040 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff5e806ac30 0x7ff5e806d0f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc009b40 tx=0x7ff5fc009190 comp rx=0 tx=0).stop 2024-09-13T16:43:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.979+0000 7ff611200640 1 -- 172.21.15.28:0/2247973040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c0c7f10 msgr2=0x7ff60c124ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.979+0000 7ff611200640 1 --2- 172.21.15.28:0/2247973040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c0c7f10 0x7ff60c124ee0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400e890 tx=0x7ff5f400ed50 comp rx=0 tx=0).stop 2024-09-13T16:43:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.979+0000 7ff611200640 1 -- 172.21.15.28:0/2247973040 shutdown_connections 2024-09-13T16:43:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.979+0000 7ff611200640 1 --2- 172.21.15.28:0/2247973040 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff5e806ac30 0x7ff5e806d0f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.979+0000 7ff611200640 1 --2- 172.21.15.28:0/2247973040 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff60c0c82f0 0x7ff60c125420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.979+0000 7ff611200640 1 --2- 172.21.15.28:0/2247973040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c0c7f10 0x7ff60c124ee0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.979+0000 7ff611200640 1 --2- 172.21.15.28:0/2247973040 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff60c0c5c00 0x7ff60c0c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.979+0000 7ff611200640 1 -- 172.21.15.28:0/2247973040 >> 172.21.15.28:0/2247973040 conn(0x7ff60c06af90 msgr2=0x7ff60c15b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.979+0000 7ff611200640 1 -- 172.21.15.28:0/2247973040 shutdown_connections 2024-09-13T16:43:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:57.979+0000 7ff611200640 1 -- 172.21.15.28:0/2247973040 wait complete. 2024-09-13T16:43:57.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T16:43:58.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.155+0000 7fbaa9800640 1 Processor -- start 2024-09-13T16:43:58.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.155+0000 7fbaa9800640 1 -- start start 2024-09-13T16:43:58.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.156+0000 7fbaa9800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbaa4071c00 0x7fbaa4159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:58.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.156+0000 7fbaa9800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa40c1f00 con 0x7fbaa4071820 2024-09-13T16:43:58.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.156+0000 7fbaa9800640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa4159c80 con 0x7fbaa414f580 2024-09-13T16:43:58.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.156+0000 7fbaa9800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa40c33f0 con 0x7fbaa4071c00 2024-09-13T16:43:58.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.156+0000 7fba9be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbaa4071c00 0x7fbaa4159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:58.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.156+0000 7fba9be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbaa4071c00 0x7fbaa4159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47514/0 (socket says 172.21.15.28:47514) 2024-09-13T16:43:58.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.156+0000 7fba9be00640 1 -- 172.21.15.28:0/438168547 learned_addr learned my addr 172.21.15.28:0/438168547 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:58.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.156+0000 7fba9be00640 1 -- 172.21.15.28:0/438168547 >> v1:172.21.15.191:6789/0 conn(0x7fbaa414f580 legacy=0x7fbaa40707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:58.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.157+0000 7fba9be00640 1 -- 172.21.15.28:0/438168547 >> v1:172.21.15.28:6789/0 conn(0x7fbaa4071820 legacy=0x7fbaa40c17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:58.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.157+0000 7fbaa3400640 1 -- 172.21.15.28:0/438168547 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248881948 0 0) 0x7fbaa40c1f00 con 0x7fbaa4071820 2024-09-13T16:43:58.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.157+0000 7fba9be00640 1 -- 172.21.15.28:0/438168547 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa40c35f0 con 0x7fbaa4071c00 2024-09-13T16:43:58.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.158+0000 7fba9be00640 1 --2- 172.21.15.28:0/438168547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa4071c00 0x7fbaa4159740 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009870 tx=0x7fba8c02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a7761affd800c25 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:58.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.158+0000 7fbaa3400640 1 -- 172.21.15.28:0/438168547 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c031710 con 0x7fbaa4071c00 2024-09-13T16:43:58.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.158+0000 7fbaa3400640 1 -- 172.21.15.28:0/438168547 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8c02f050 con 0x7fbaa4071c00 2024-09-13T16:43:58.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.158+0000 7fbaa3400640 1 -- 172.21.15.28:0/438168547 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c031a90 con 0x7fbaa4071c00 2024-09-13T16:43:58.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.158+0000 7fbaa9800640 1 -- 172.21.15.28:0/438168547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa4071c00 msgr2=0x7fbaa4159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:58.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.158+0000 7fbaa9800640 1 --2- 172.21.15.28:0/438168547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa4071c00 0x7fbaa4159740 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009870 tx=0x7fba8c02c5d0 comp rx=0 tx=0).stop 2024-09-13T16:43:58.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.158+0000 7fbaa9800640 1 -- 172.21.15.28:0/438168547 shutdown_connections 2024-09-13T16:43:58.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.159+0000 7fbaa9800640 1 --2- 172.21.15.28:0/438168547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa4071c00 0x7fbaa4159740 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:58.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.159+0000 7fbaa9800640 1 -- 172.21.15.28:0/438168547 >> 172.21.15.28:0/438168547 conn(0x7fbaa4069e80 msgr2=0x7fbaa406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:58.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.159+0000 7fbaa9800640 1 -- 172.21.15.28:0/438168547 shutdown_connections 2024-09-13T16:43:58.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.159+0000 7fbaa9800640 1 -- 172.21.15.28:0/438168547 wait complete. 2024-09-13T16:43:58.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.160+0000 7fbaa9800640 1 Processor -- start 2024-09-13T16:43:58.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.160+0000 7fbaa9800640 1 -- start start 2024-09-13T16:43:58.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.161+0000 7fbaa9800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbaa4071820 0x7fbaa41502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:58.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.161+0000 7fbaa9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa4071c00 0x7fbaa41edb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:58.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.161+0000 7fbaa9800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbaa414f580 0x7fbaa4158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:58.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.161+0000 7fbaa9800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa41ed190 con 0x7fbaa4071c00 2024-09-13T16:43:58.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.161+0000 7fbaa9800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa41ed6f0 con 0x7fbaa4071820 2024-09-13T16:43:58.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.161+0000 7fbaa9800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa4148130 con 0x7fbaa414f580 2024-09-13T16:43:58.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.161+0000 7fba9be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa4071c00 0x7fbaa41edb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:58.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.161+0000 7fba9be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa4071c00 0x7fbaa41edb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47528/0 (socket says 172.21.15.28:47528) 2024-09-13T16:43:58.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.161+0000 7fba9be00640 1 -- 172.21.15.28:0/1236115942 learned_addr learned my addr 172.21.15.28:0/1236115942 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:58.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.162+0000 7fbaa3e00640 1 --2- 172.21.15.28:0/1236115942 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbaa4071820 0x7fbaa41502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:58.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.162+0000 7fbaa8e00640 1 --2- 172.21.15.28:0/1236115942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbaa414f580 0x7fbaa4158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:58.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.162+0000 7fba9be00640 1 -- 172.21.15.28:0/1236115942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbaa414f580 msgr2=0x7fbaa4158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:58.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.162+0000 7fba9be00640 1 --2- 172.21.15.28:0/1236115942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbaa414f580 0x7fbaa4158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:58.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.162+0000 7fba9be00640 1 -- 172.21.15.28:0/1236115942 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbaa4071820 msgr2=0x7fbaa41502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:58.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.162+0000 7fba9be00640 1 --2- 172.21.15.28:0/1236115942 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbaa4071820 0x7fbaa41502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:58.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.162+0000 7fba9be00640 1 -- 172.21.15.28:0/1236115942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c0094d0 con 0x7fbaa4071c00 2024-09-13T16:43:58.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.162+0000 7fbaa3e00640 1 --2- 172.21.15.28:0/1236115942 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbaa4071820 0x7fbaa41502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:43:58.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.162+0000 7fbaa8e00640 1 --2- 172.21.15.28:0/1236115942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbaa414f580 0x7fbaa4158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:43:58.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.162+0000 7fba9be00640 1 --2- 172.21.15.28:0/1236115942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa4071c00 0x7fbaa41edb80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fba8c000c00 tx=0x7fba8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:58.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.162+0000 7fbaa0c00640 1 -- 172.21.15.28:0/1236115942 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c033890 con 0x7fbaa4071c00 2024-09-13T16:43:58.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.162+0000 7fbaa0c00640 1 -- 172.21.15.28:0/1236115942 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8c033a10 con 0x7fbaa4071c00 2024-09-13T16:43:58.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.162+0000 7fbaa9800640 1 -- 172.21.15.28:0/1236115942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa41ee0c0 con 0x7fbaa4071c00 2024-09-13T16:43:58.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.162+0000 7fbaa0c00640 1 -- 172.21.15.28:0/1236115942 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c033dc0 con 0x7fbaa4071c00 2024-09-13T16:43:58.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.163+0000 7fbaa9800640 1 -- 172.21.15.28:0/1236115942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa4148690 con 0x7fbaa4071c00 2024-09-13T16:43:58.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.164+0000 7fbaa9800640 1 -- 172.21.15.28:0/1236115942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa414ba60 con 0x7fbaa4071c00 2024-09-13T16:43:58.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.164+0000 7fbaa0c00640 1 -- 172.21.15.28:0/1236115942 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fba8c03a070 con 0x7fbaa4071c00 2024-09-13T16:43:58.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.165+0000 7fbaa0c00640 1 --2- 172.21.15.28:0/1236115942 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fba8006ac30 0x7fba8006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:58.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.165+0000 7fbaa0c00640 1 -- 172.21.15.28:0/1236115942 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba8c0aa5c0 con 0x7fbaa4071c00 2024-09-13T16:43:58.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.166+0000 7fbaa3e00640 1 --2- 172.21.15.28:0/1236115942 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fba8006ac30 0x7fba8006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:58.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.166+0000 7fbaa3e00640 1 --2- 172.21.15.28:0/1236115942 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fba8006ac30 0x7fba8006d0f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fba94000f60 tx=0x7fba94009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:58.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.169+0000 7fbaa0c00640 1 -- 172.21.15.28:0/1236115942 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fba8c075830 con 0x7fbaa4071c00 2024-09-13T16:43:58.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.384+0000 7fbaa9800640 1 -- 172.21.15.28:0/1236115942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbaa4159740 con 0x7fbaa4071c00 2024-09-13T16:43:58.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.977+0000 7fbaa0c00640 1 -- 172.21.15.28:0/1236115942 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7fba8c078e50 con 0x7fbaa4071c00 2024-09-13T16:43:58.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.980+0000 7fbaa9800640 1 -- 172.21.15.28:0/1236115942 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fba8006ac30 msgr2=0x7fba8006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:58.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.980+0000 7fbaa9800640 1 --2- 172.21.15.28:0/1236115942 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fba8006ac30 0x7fba8006d0f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fba94000f60 tx=0x7fba94009040 comp rx=0 tx=0).stop 2024-09-13T16:43:58.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.980+0000 7fbaa9800640 1 -- 172.21.15.28:0/1236115942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa4071c00 msgr2=0x7fbaa41edb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:58.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.980+0000 7fbaa9800640 1 --2- 172.21.15.28:0/1236115942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa4071c00 0x7fbaa41edb80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fba8c000c00 tx=0x7fba8c0046c0 comp rx=0 tx=0).stop 2024-09-13T16:43:58.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.981+0000 7fbaa9800640 1 -- 172.21.15.28:0/1236115942 shutdown_connections 2024-09-13T16:43:58.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.981+0000 7fbaa9800640 1 --2- 172.21.15.28:0/1236115942 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fba8006ac30 0x7fba8006d0f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:58.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.981+0000 7fbaa9800640 1 --2- 172.21.15.28:0/1236115942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbaa414f580 0x7fbaa4158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:58.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.981+0000 7fbaa9800640 1 --2- 172.21.15.28:0/1236115942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaa4071c00 0x7fbaa41edb80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:58.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.981+0000 7fbaa9800640 1 --2- 172.21.15.28:0/1236115942 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbaa4071820 0x7fbaa41502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:58.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.981+0000 7fbaa9800640 1 -- 172.21.15.28:0/1236115942 >> 172.21.15.28:0/1236115942 conn(0x7fbaa4069e80 msgr2=0x7fbaa414e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:58.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.981+0000 7fbaa9800640 1 -- 172.21.15.28:0/1236115942 shutdown_connections 2024-09-13T16:43:58.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:58.981+0000 7fbaa9800640 1 -- 172.21.15.28:0/1236115942 wait complete. 2024-09-13T16:43:59.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-09-13T16:43:59.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.162+0000 7fc847400640 1 Processor -- start 2024-09-13T16:43:59.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.162+0000 7fc847400640 1 -- start start 2024-09-13T16:43:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.164+0000 7fc847400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc840152d50 0x7fc8400c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.164+0000 7fc847400640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc8400c5340 0x7fc84006d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.164+0000 7fc847400640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8400c6cf0 con 0x7fc8400c5740 2024-09-13T16:43:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.164+0000 7fc847400640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc84015c080 con 0x7fc8400c5340 2024-09-13T16:43:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.164+0000 7fc847400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc84015cbc0 con 0x7fc840152d50 2024-09-13T16:43:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.164+0000 7fc844a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc840152d50 0x7fc8400c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.164+0000 7fc844a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc840152d50 0x7fc8400c67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37190/0 (socket says 172.21.15.28:37190) 2024-09-13T16:43:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.164+0000 7fc844a00640 1 -- 172.21.15.28:0/1068880037 learned_addr learned my addr 172.21.15.28:0/1068880037 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.164+0000 7fc837e00640 1 --2- 172.21.15.28:0/1068880037 >> v2:172.21.15.191:3300/0 conn(0x7fc8400c5340 0x7fc84006d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:59.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.164+0000 7fc844a00640 1 -- 172.21.15.28:0/1068880037 >> v2:172.21.15.191:3300/0 conn(0x7fc8400c5340 msgr2=0x7fc84006d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:43:59.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.164+0000 7fc844a00640 1 --2- 172.21.15.28:0/1068880037 >> v2:172.21.15.191:3300/0 conn(0x7fc8400c5340 0x7fc84006d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:59.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.164+0000 7fc844a00640 1 -- 172.21.15.28:0/1068880037 >> v1:172.21.15.191:6789/0 conn(0x7fc8400c5740 legacy=0x7fc84006fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:59.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.164+0000 7fc844a00640 1 -- 172.21.15.28:0/1068880037 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8400c5d10 con 0x7fc840152d50 2024-09-13T16:43:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.164+0000 7fc83fe00640 1 -- 172.21.15.28:0/1068880037 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550330346 0 0) 0x7fc838001490 con 0x7fc8400c5740 2024-09-13T16:43:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.165+0000 7fc844a00640 1 --2- 172.21.15.28:0/1068880037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc840152d50 0x7fc8400c67b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc830009870 tx=0x7fc83002a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d15f32f8bc9d8e3b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.165+0000 7fc83fe00640 1 -- 172.21.15.28:0/1068880037 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc830032640 con 0x7fc840152d50 2024-09-13T16:43:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.165+0000 7fc83fe00640 1 -- 172.21.15.28:0/1068880037 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8300327a0 con 0x7fc840152d50 2024-09-13T16:43:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.165+0000 7fc83fe00640 1 -- 172.21.15.28:0/1068880037 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc830032b30 con 0x7fc840152d50 2024-09-13T16:43:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.165+0000 7fc847400640 1 -- 172.21.15.28:0/1068880037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc840152d50 msgr2=0x7fc8400c67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.165+0000 7fc847400640 1 --2- 172.21.15.28:0/1068880037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc840152d50 0x7fc8400c67b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc830009870 tx=0x7fc83002a300 comp rx=0 tx=0).stop 2024-09-13T16:43:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.165+0000 7fc847400640 1 -- 172.21.15.28:0/1068880037 shutdown_connections 2024-09-13T16:43:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.165+0000 7fc847400640 1 --2- 172.21.15.28:0/1068880037 >> v2:172.21.15.191:3300/0 conn(0x7fc8400c5340 0x7fc84006d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.165+0000 7fc847400640 1 --2- 172.21.15.28:0/1068880037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc840152d50 0x7fc8400c67b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.165+0000 7fc847400640 1 -- 172.21.15.28:0/1068880037 >> 172.21.15.28:0/1068880037 conn(0x7fc8400ca6b0 msgr2=0x7fc8400caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.166+0000 7fc847400640 1 -- 172.21.15.28:0/1068880037 shutdown_connections 2024-09-13T16:43:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.166+0000 7fc847400640 1 -- 172.21.15.28:0/1068880037 wait complete. 2024-09-13T16:43:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.167+0000 7fc847400640 1 Processor -- start 2024-09-13T16:43:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.167+0000 7fc847400640 1 -- start start 2024-09-13T16:43:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.168+0000 7fc847400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8400c5340 0x7fc840156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.168+0000 7fc847400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8400c5740 0x7fc8401eede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.168+0000 7fc847400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc840152d50 0x7fc840149190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.168+0000 7fc847400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8401571c0 con 0x7fc8400c5740 2024-09-13T16:43:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.168+0000 7fc847400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84015adf0 con 0x7fc8400c5340 2024-09-13T16:43:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.168+0000 7fc847400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8401f1a70 con 0x7fc840152d50 2024-09-13T16:43:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.168+0000 7fc844a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8400c5340 0x7fc840156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.168+0000 7fc837e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8400c5740 0x7fc8401eede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.168+0000 7fc837e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8400c5740 0x7fc8401eede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41332/0 (socket says 172.21.15.28:41332) 2024-09-13T16:43:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.168+0000 7fc844a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8400c5340 0x7fc840156770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60458/0 (socket says 172.21.15.28:60458) 2024-09-13T16:43:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.168+0000 7fc837e00640 1 -- 172.21.15.28:0/454390585 learned_addr learned my addr 172.21.15.28:0/454390585 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:43:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.168+0000 7fc845400640 1 --2- 172.21.15.28:0/454390585 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc840152d50 0x7fc840149190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.169+0000 7fc837e00640 1 -- 172.21.15.28:0/454390585 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc840152d50 msgr2=0x7fc840149190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.169+0000 7fc837e00640 1 --2- 172.21.15.28:0/454390585 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc840152d50 0x7fc840149190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.169+0000 7fc837e00640 1 -- 172.21.15.28:0/454390585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8400c5340 msgr2=0x7fc840156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.169+0000 7fc837e00640 1 --2- 172.21.15.28:0/454390585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8400c5340 0x7fc840156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.169+0000 7fc837e00640 1 -- 172.21.15.28:0/454390585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8300094d0 con 0x7fc8400c5740 2024-09-13T16:43:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.169+0000 7fc837e00640 1 --2- 172.21.15.28:0/454390585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8400c5740 0x7fc8401eede0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc82800e9e0 tx=0x7fc82800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.169+0000 7fc83d600640 1 -- 172.21.15.28:0/454390585 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc828009670 con 0x7fc8400c5740 2024-09-13T16:43:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.169+0000 7fc83d600640 1 -- 172.21.15.28:0/454390585 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82800c970 con 0x7fc8400c5740 2024-09-13T16:43:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.169+0000 7fc847400640 1 -- 172.21.15.28:0/454390585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc84015b8e0 con 0x7fc8400c5740 2024-09-13T16:43:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.169+0000 7fc83d600640 1 -- 172.21.15.28:0/454390585 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82800cd20 con 0x7fc8400c5740 2024-09-13T16:43:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.169+0000 7fc847400640 1 -- 172.21.15.28:0/454390585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84015b320 con 0x7fc8400c5740 2024-09-13T16:43:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.170+0000 7fc847400640 1 -- 172.21.15.28:0/454390585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8400c67b0 con 0x7fc8400c5740 2024-09-13T16:43:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.171+0000 7fc83d600640 1 -- 172.21.15.28:0/454390585 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc828013470 con 0x7fc8400c5740 2024-09-13T16:43:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.172+0000 7fc83d600640 1 --2- 172.21.15.28:0/454390585 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc81c06ac00 0x7fc81c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:43:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.172+0000 7fc83d600640 1 -- 172.21.15.28:0/454390585 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8280894e0 con 0x7fc8400c5740 2024-09-13T16:43:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.173+0000 7fc844a00640 1 --2- 172.21.15.28:0/454390585 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc81c06ac00 0x7fc81c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:43:59.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.173+0000 7fc844a00640 1 --2- 172.21.15.28:0/454390585 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc81c06ac00 0x7fc81c06d0c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc830034000 tx=0x7fc83002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:43:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.177+0000 7fc83d600640 1 -- 172.21.15.28:0/454390585 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc8280546e0 con 0x7fc8400c5740 2024-09-13T16:43:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.392+0000 7fc847400640 1 -- 172.21.15.28:0/454390585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fc8401f1bb0 con 0x7fc8400c5740 2024-09-13T16:43:59.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.989+0000 7fc83d600640 1 -- 172.21.15.28:0/454390585 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) v1 ==== 115+0+0 (secure 0 0 0) 0x7fc828057d00 con 0x7fc8400c5740 2024-09-13T16:43:59.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.992+0000 7fc847400640 1 -- 172.21.15.28:0/454390585 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc81c06ac00 msgr2=0x7fc81c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:59.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.992+0000 7fc847400640 1 --2- 172.21.15.28:0/454390585 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc81c06ac00 0x7fc81c06d0c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc830034000 tx=0x7fc83002e000 comp rx=0 tx=0).stop 2024-09-13T16:43:59.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.992+0000 7fc847400640 1 -- 172.21.15.28:0/454390585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8400c5740 msgr2=0x7fc8401eede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:43:59.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.992+0000 7fc847400640 1 --2- 172.21.15.28:0/454390585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8400c5740 0x7fc8401eede0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc82800e9e0 tx=0x7fc82800eea0 comp rx=0 tx=0).stop 2024-09-13T16:43:59.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.992+0000 7fc847400640 1 -- 172.21.15.28:0/454390585 shutdown_connections 2024-09-13T16:43:59.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.992+0000 7fc847400640 1 --2- 172.21.15.28:0/454390585 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc81c06ac00 0x7fc81c06d0c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:59.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.992+0000 7fc847400640 1 --2- 172.21.15.28:0/454390585 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc840152d50 0x7fc840149190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:59.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.992+0000 7fc847400640 1 --2- 172.21.15.28:0/454390585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8400c5740 0x7fc8401eede0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:59.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.992+0000 7fc847400640 1 --2- 172.21.15.28:0/454390585 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc8400c5340 0x7fc840156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:43:59.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.992+0000 7fc847400640 1 -- 172.21.15.28:0/454390585 >> 172.21.15.28:0/454390585 conn(0x7fc8400ca6b0 msgr2=0x7fc84006d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:43:59.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.992+0000 7fc847400640 1 -- 172.21.15.28:0/454390585 shutdown_connections 2024-09-13T16:43:59.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:43:59.992+0000 7fc847400640 1 -- 172.21.15.28:0/454390585 wait complete. 2024-09-13T16:44:00.012 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T16:44:00.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:44:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.170+0000 7f29bce00640 1 Processor -- start 2024-09-13T16:44:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.170+0000 7f29bce00640 1 -- start start 2024-09-13T16:44:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.171+0000 7f29bce00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29b806acd0 0x7f29b806b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.171+0000 7f29bce00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b80cadc0 con 0x7f29b815cc50 2024-09-13T16:44:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.171+0000 7f29bce00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b806bc40 con 0x7f29b8152d10 2024-09-13T16:44:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.171+0000 7f29bce00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b80ca4d0 con 0x7f29b806acd0 2024-09-13T16:44:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.171+0000 7f29afe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f29b815cc50 0x7f29b815c0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59454/0 (socket says 172.21.15.28:59454) 2024-09-13T16:44:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.171+0000 7f29afe00640 1 -- 172.21.15.28:0/560631586 learned_addr learned my addr 172.21.15.28:0/560631586 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.171+0000 7f29b6800640 1 --2- 172.21.15.28:0/560631586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29b806acd0 0x7f29b806b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.172+0000 7f29b5400640 1 -- 172.21.15.28:0/560631586 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779024899 0 0) 0x7f29b80cadc0 con 0x7f29b815cc50 2024-09-13T16:44:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.172+0000 7f29b5400640 1 -- 172.21.15.28:0/560631586 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f299c003520 con 0x7f29b815cc50 2024-09-13T16:44:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.172+0000 7f29b5400640 1 -- 172.21.15.28:0/560631586 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724169864 0 0) 0x7f29b806bc40 con 0x7f29b8152d10 2024-09-13T16:44:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.172+0000 7f29b5400640 1 -- 172.21.15.28:0/560631586 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29b80cadc0 con 0x7f29b8152d10 2024-09-13T16:44:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.172+0000 7f29b6800640 1 -- 172.21.15.28:0/560631586 >> v1:172.21.15.191:6789/0 conn(0x7f29b8152d10 legacy=0x7f29b80cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.172+0000 7f29b6800640 1 -- 172.21.15.28:0/560631586 >> v1:172.21.15.28:6789/0 conn(0x7f29b815cc50 legacy=0x7f29b815c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.172+0000 7f29b6800640 1 -- 172.21.15.28:0/560631586 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b80cb960 con 0x7f29b806acd0 2024-09-13T16:44:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.173+0000 7f29b6800640 1 --2- 172.21.15.28:0/560631586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29b806acd0 0x7f29b806b120 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009df0 tx=0x7f29b002a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b2a0b111dd1e78c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.173+0000 7f29b5400640 1 -- 172.21.15.28:0/560631586 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b0007740 con 0x7f29b806acd0 2024-09-13T16:44:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.173+0000 7f29b5400640 1 -- 172.21.15.28:0/560631586 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b002d050 con 0x7f29b806acd0 2024-09-13T16:44:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.173+0000 7f29b5400640 1 -- 172.21.15.28:0/560631586 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b0007ac0 con 0x7f29b806acd0 2024-09-13T16:44:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.174+0000 7f29bce00640 1 -- 172.21.15.28:0/560631586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29b806acd0 msgr2=0x7f29b806b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.174+0000 7f29bce00640 1 --2- 172.21.15.28:0/560631586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29b806acd0 0x7f29b806b120 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009df0 tx=0x7f29b002a710 comp rx=0 tx=0).stop 2024-09-13T16:44:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.174+0000 7f29bce00640 1 -- 172.21.15.28:0/560631586 shutdown_connections 2024-09-13T16:44:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.174+0000 7f29bce00640 1 --2- 172.21.15.28:0/560631586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29b806acd0 0x7f29b806b120 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.174+0000 7f29bce00640 1 -- 172.21.15.28:0/560631586 >> 172.21.15.28:0/560631586 conn(0x7f29b8065c50 msgr2=0x7f29b8066060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.174+0000 7f29bce00640 1 -- 172.21.15.28:0/560631586 shutdown_connections 2024-09-13T16:44:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.174+0000 7f29bce00640 1 -- 172.21.15.28:0/560631586 wait complete. 2024-09-13T16:44:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.175+0000 7f29bce00640 1 Processor -- start 2024-09-13T16:44:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.176+0000 7f29bce00640 1 -- start start 2024-09-13T16:44:00.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.176+0000 7f29bce00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29b806acd0 0x7f29b814c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:00.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.176+0000 7f29bce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b8152d10 0x7f29b81e4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:00.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.176+0000 7f29bce00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29b815cc50 0x7f29b80c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:00.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.176+0000 7f29bce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b81e5340 con 0x7f29b8152d10 2024-09-13T16:44:00.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.176+0000 7f29bce00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b81ee360 con 0x7f29b815cc50 2024-09-13T16:44:00.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.177+0000 7f29bce00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b81e5560 con 0x7f29b806acd0 2024-09-13T16:44:00.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.177+0000 7f29afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b8152d10 0x7f29b81e4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:00.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.177+0000 7f29b5e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29b806acd0 0x7f29b814c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:00.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.177+0000 7f29b6800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29b815cc50 0x7f29b80c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:00.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.177+0000 7f29afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b8152d10 0x7f29b81e4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41336/0 (socket says 172.21.15.28:41336) 2024-09-13T16:44:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.177+0000 7f29afe00640 1 -- 172.21.15.28:0/2897757088 learned_addr learned my addr 172.21.15.28:0/2897757088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.177+0000 7f29afe00640 1 -- 172.21.15.28:0/2897757088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29b806acd0 msgr2=0x7f29b814c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.177+0000 7f29afe00640 1 --2- 172.21.15.28:0/2897757088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29b806acd0 0x7f29b814c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.177+0000 7f29afe00640 1 -- 172.21.15.28:0/2897757088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29b815cc50 msgr2=0x7f29b80c5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.177+0000 7f29afe00640 1 --2- 172.21.15.28:0/2897757088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29b815cc50 0x7f29b80c5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.177+0000 7f29afe00640 1 -- 172.21.15.28:0/2897757088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b0009a70 con 0x7f29b8152d10 2024-09-13T16:44:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.177+0000 7f29b6800640 1 --2- 172.21.15.28:0/2897757088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29b815cc50 0x7f29b80c5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.178+0000 7f29afe00640 1 --2- 172.21.15.28:0/2897757088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b8152d10 0x7f29b81e4e00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f29a000c500 tx=0x7f29a000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.178+0000 7f29ae000640 1 -- 172.21.15.28:0/2897757088 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0005ce0 con 0x7f29b8152d10 2024-09-13T16:44:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.178+0000 7f29ae000640 1 -- 172.21.15.28:0/2897757088 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a0005e60 con 0x7f29b8152d10 2024-09-13T16:44:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.178+0000 7f29ae000640 1 -- 172.21.15.28:0/2897757088 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0007330 con 0x7f29b8152d10 2024-09-13T16:44:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.178+0000 7f29bce00640 1 -- 172.21.15.28:0/2897757088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b81ee620 con 0x7f29b8152d10 2024-09-13T16:44:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.178+0000 7f29bce00640 1 -- 172.21.15.28:0/2897757088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b81eebe0 con 0x7f29b8152d10 2024-09-13T16:44:00.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.179+0000 7f29bce00640 1 -- 172.21.15.28:0/2897757088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b80c34d0 con 0x7f29b8152d10 2024-09-13T16:44:00.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.180+0000 7f29ae000640 1 -- 172.21.15.28:0/2897757088 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f29a0007490 con 0x7f29b8152d10 2024-09-13T16:44:00.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.181+0000 7f29ae000640 1 --2- 172.21.15.28:0/2897757088 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f299406ac00 0x7f299406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.182+0000 7f29ae000640 1 -- 172.21.15.28:0/2897757088 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29a0088920 con 0x7f29b8152d10 2024-09-13T16:44:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.182+0000 7f29b5e00640 1 --2- 172.21.15.28:0/2897757088 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f299406ac00 0x7f299406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.182+0000 7f29b5e00640 1 --2- 172.21.15.28:0/2897757088 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f299406ac00 0x7f299406d0c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f29a80011e0 tx=0x7f29a8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:00.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.186+0000 7f29ae000640 1 -- 172.21.15.28:0/2897757088 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f29a0053c60 con 0x7f29b8152d10 2024-09-13T16:44:00.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.400+0000 7f29bce00640 1 -- 172.21.15.28:0/2897757088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29b80cf560 con 0x7f29b8152d10 2024-09-13T16:44:00.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.401+0000 7f29ae000640 1 -- 172.21.15.28:0/2897757088 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6246 (secure 0 0 0) 0x7f29a0017020 con 0x7f29b8152d10 2024-09-13T16:44:00.403 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:00.403 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4227,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/355570369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":355570369},{"type":"v1","addr":"172.21.15.28:6835","nonce":355570369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4232,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/2366383263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6837","nonce":2366383263},{"type":"v1","addr":"172.21.15.191:6839","nonce":2366383263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6838/2702860767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2702860767},{"type":"v1","addr":"172.21.15.191:6838","nonce":2702860767}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2240589617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2240589617},{"type":"v1","addr":"172.21.15.28:6833","nonce":2240589617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4250,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/4068563017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":4068563017},{"type":"v1","addr":"172.21.15.28:6837","nonce":4068563017}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:43:54.969699+0000","modified":"2024-09-13T16:43:59.979988+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0],"up":{"mds_0":4258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4258":{"gid":4258,"name":"d","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.191:6835/704943585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":704943585},{"type":"v1","addr":"172.21.15.191:6835","nonce":704943585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T16:44:00.403 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-13T16:44:00.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.403+0000 7f29bce00640 1 -- 172.21.15.28:0/2897757088 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f299406ac00 msgr2=0x7f299406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.403+0000 7f29bce00640 1 --2- 172.21.15.28:0/2897757088 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f299406ac00 0x7f299406d0c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f29a80011e0 tx=0x7f29a8009040 comp rx=0 tx=0).stop 2024-09-13T16:44:00.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.403+0000 7f29bce00640 1 -- 172.21.15.28:0/2897757088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b8152d10 msgr2=0x7f29b81e4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.403+0000 7f29bce00640 1 --2- 172.21.15.28:0/2897757088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b8152d10 0x7f29b81e4e00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f29a000c500 tx=0x7f29a000c9c0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.404+0000 7f29bce00640 1 -- 172.21.15.28:0/2897757088 shutdown_connections 2024-09-13T16:44:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.404+0000 7f29bce00640 1 --2- 172.21.15.28:0/2897757088 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f299406ac00 0x7f299406d0c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.404+0000 7f29bce00640 1 --2- 172.21.15.28:0/2897757088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29b815cc50 0x7f29b80c5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.404+0000 7f29bce00640 1 --2- 172.21.15.28:0/2897757088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b8152d10 0x7f29b81e4e00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.404+0000 7f29bce00640 1 --2- 172.21.15.28:0/2897757088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29b806acd0 0x7f29b814c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.404+0000 7f29bce00640 1 -- 172.21.15.28:0/2897757088 >> 172.21.15.28:0/2897757088 conn(0x7f29b8065c50 msgr2=0x7f29b80c9300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.404+0000 7f29bce00640 1 -- 172.21.15.28:0/2897757088 shutdown_connections 2024-09-13T16:44:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.404+0000 7f29bce00640 1 -- 172.21.15.28:0/2897757088 wait complete. 2024-09-13T16:44:00.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:44:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.573+0000 7ff6d7c00640 1 Processor -- start 2024-09-13T16:44:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.573+0000 7ff6d7c00640 1 -- start start 2024-09-13T16:44:00.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.575+0000 7ff6d7c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff6d00c3670 0x7ff6d00c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:00.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.575+0000 7ff6d7c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d014ca00 con 0x7ff6d00c1ea0 2024-09-13T16:44:00.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.575+0000 7ff6d7c00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d0155990 con 0x7ff6d00c5bd0 2024-09-13T16:44:00.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.575+0000 7ff6d7c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d00c2dc0 con 0x7ff6d00c3670 2024-09-13T16:44:00.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.575+0000 7ff6c7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff6d00c3670 0x7ff6d00c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:00.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.575+0000 7ff6c7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff6d00c3670 0x7ff6d00c1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41344/0 (socket says 172.21.15.28:41344) 2024-09-13T16:44:00.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.575+0000 7ff6c7e00640 1 -- 172.21.15.28:0/1666761357 learned_addr learned my addr 172.21.15.28:0/1666761357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:00.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.575+0000 7ff6d5c00640 1 --1- 172.21.15.28:0/1666761357 >> v1:172.21.15.28:6789/0 conn(0x7ff6d00c1ea0 0x7ff6d00c6ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59456/0 (socket says 172.21.15.28:59456) 2024-09-13T16:44:00.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.575+0000 7ff6cfe00640 1 -- 172.21.15.28:0/1666761357 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844540997 0 0) 0x7ff6d0155990 con 0x7ff6d00c5bd0 2024-09-13T16:44:00.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.575+0000 7ff6c7e00640 1 -- 172.21.15.28:0/1666761357 >> v1:172.21.15.191:6789/0 conn(0x7ff6d00c5bd0 legacy=0x7ff6d0156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.575+0000 7ff6c7e00640 1 -- 172.21.15.28:0/1666761357 >> v1:172.21.15.28:6789/0 conn(0x7ff6d00c1ea0 legacy=0x7ff6d00c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.575+0000 7ff6c7e00640 1 -- 172.21.15.28:0/1666761357 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d00c3b60 con 0x7ff6d00c3670 2024-09-13T16:44:00.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.576+0000 7ff6c7e00640 1 --2- 172.21.15.28:0/1666761357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d00c3670 0x7ff6d00c1960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff6b800bb10 tx=0x7ff6b802ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=717ea2719bbcc5f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.577+0000 7ff6cfe00640 1 -- 172.21.15.28:0/1666761357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8030820 con 0x7ff6d00c3670 2024-09-13T16:44:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.577+0000 7ff6cfe00640 1 -- 172.21.15.28:0/1666761357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b8030980 con 0x7ff6d00c3670 2024-09-13T16:44:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.577+0000 7ff6cfe00640 1 -- 172.21.15.28:0/1666761357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8030d10 con 0x7ff6d00c3670 2024-09-13T16:44:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.577+0000 7ff6d7c00640 1 -- 172.21.15.28:0/1666761357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d00c3670 msgr2=0x7ff6d00c1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.577+0000 7ff6d7c00640 1 --2- 172.21.15.28:0/1666761357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d00c3670 0x7ff6d00c1960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff6b800bb10 tx=0x7ff6b802ca60 comp rx=0 tx=0).stop 2024-09-13T16:44:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.577+0000 7ff6d7c00640 1 -- 172.21.15.28:0/1666761357 shutdown_connections 2024-09-13T16:44:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.577+0000 7ff6d7c00640 1 --2- 172.21.15.28:0/1666761357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d00c3670 0x7ff6d00c1960 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.577+0000 7ff6d7c00640 1 -- 172.21.15.28:0/1666761357 >> 172.21.15.28:0/1666761357 conn(0x7ff6d0066040 msgr2=0x7ff6d0069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.578+0000 7ff6d7c00640 1 -- 172.21.15.28:0/1666761357 shutdown_connections 2024-09-13T16:44:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.578+0000 7ff6d7c00640 1 -- 172.21.15.28:0/1666761357 wait complete. 2024-09-13T16:44:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.579+0000 7ff6d7c00640 1 Processor -- start 2024-09-13T16:44:00.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.579+0000 7ff6d7c00640 1 -- start start 2024-09-13T16:44:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.580+0000 7ff6d7c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6d00c1ea0 0x7ff6d0144160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.580+0000 7ff6d7c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6d00c3670 0x7ff6d01ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.580+0000 7ff6d7c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d00c5bd0 0x7ff6d00bd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.580+0000 7ff6d7c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d00bca20 con 0x7ff6d00c5bd0 2024-09-13T16:44:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.580+0000 7ff6d7c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d0146c50 con 0x7ff6d00c3670 2024-09-13T16:44:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.580+0000 7ff6d7c00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d00bcfa0 con 0x7ff6d00c1ea0 2024-09-13T16:44:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.580+0000 7ff6d5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d00c5bd0 0x7ff6d00bd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.581+0000 7ff6d5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d00c5bd0 0x7ff6d00bd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41346/0 (socket says 172.21.15.28:41346) 2024-09-13T16:44:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.581+0000 7ff6d5c00640 1 -- 172.21.15.28:0/1336083969 learned_addr learned my addr 172.21.15.28:0/1336083969 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.581+0000 7ff6c7e00640 1 --2- 172.21.15.28:0/1336083969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6d00c3670 0x7ff6d01ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.581+0000 7ff6d5200640 1 --2- 172.21.15.28:0/1336083969 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6d00c1ea0 0x7ff6d0144160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.581+0000 7ff6d5c00640 1 -- 172.21.15.28:0/1336083969 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6d00c1ea0 msgr2=0x7ff6d0144160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.581+0000 7ff6d5c00640 1 --2- 172.21.15.28:0/1336083969 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6d00c1ea0 0x7ff6d0144160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.581+0000 7ff6d5c00640 1 -- 172.21.15.28:0/1336083969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6d00c3670 msgr2=0x7ff6d01ee6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.581+0000 7ff6d5c00640 1 --2- 172.21.15.28:0/1336083969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6d00c3670 0x7ff6d01ee6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.581+0000 7ff6d5c00640 1 -- 172.21.15.28:0/1336083969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b800b770 con 0x7ff6d00c5bd0 2024-09-13T16:44:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.581+0000 7ff6d5c00640 1 --2- 172.21.15.28:0/1336083969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d00c5bd0 0x7ff6d00bd850 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c7b0 tx=0x7ff6c800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.581+0000 7ff6cd600640 1 -- 172.21.15.28:0/1336083969 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c800a640 con 0x7ff6d00c5bd0 2024-09-13T16:44:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.581+0000 7ff6cd600640 1 -- 172.21.15.28:0/1336083969 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c800a7a0 con 0x7ff6d00c5bd0 2024-09-13T16:44:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.581+0000 7ff6cd600640 1 -- 172.21.15.28:0/1336083969 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c800ab10 con 0x7ff6d00c5bd0 2024-09-13T16:44:00.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.582+0000 7ff6d7c00640 1 -- 172.21.15.28:0/1336083969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d01446a0 con 0x7ff6d00c5bd0 2024-09-13T16:44:00.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.582+0000 7ff6d7c00640 1 -- 172.21.15.28:0/1336083969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d0147180 con 0x7ff6d00c5bd0 2024-09-13T16:44:00.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.583+0000 7ff6d7c00640 1 -- 172.21.15.28:0/1336083969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d00c4d30 con 0x7ff6d00c5bd0 2024-09-13T16:44:00.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.584+0000 7ff6cd600640 1 -- 172.21.15.28:0/1336083969 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff6c800ac70 con 0x7ff6d00c5bd0 2024-09-13T16:44:00.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.585+0000 7ff6cd600640 1 --2- 172.21.15.28:0/1336083969 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff6ac06ac30 0x7ff6ac06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:00.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.585+0000 7ff6cd600640 1 -- 172.21.15.28:0/1336083969 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6c8088b40 con 0x7ff6d00c5bd0 2024-09-13T16:44:00.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.585+0000 7ff6d5200640 1 --2- 172.21.15.28:0/1336083969 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff6ac06ac30 0x7ff6ac06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:00.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.586+0000 7ff6d5200640 1 --2- 172.21.15.28:0/1336083969 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff6ac06ac30 0x7ff6ac06d0f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0000f60 tx=0x7ff6c0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:00.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.589+0000 7ff6cd600640 1 -- 172.21.15.28:0/1336083969 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff6c8053d40 con 0x7ff6d00c5bd0 2024-09-13T16:44:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.769+0000 7ff6d7c00640 1 -- 172.21.15.28:0/1336083969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6d0070680 con 0x7ff6d00c5bd0 2024-09-13T16:44:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.771+0000 7ff6cd600640 1 -- 172.21.15.28:0/1336083969 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16481 (secure 0 0 0) 0x7ff6c8017070 con 0x7ff6d00c5bd0 2024-09-13T16:44:00.772 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:00.772 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:43:55.971358+0000","last_up_change":"2024-09-13T16:43:49.476313+0000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:43:49.643606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:43:52.828510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:43:53.945874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6801","nonce":2964323435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6803","nonce":2964323435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6807","nonce":2964323435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6805","nonce":2964323435}]},"public_addr":"172.21.15.28:6801/2964323435","cluster_addr":"172.21.15.28:6803/2964323435","heartbeat_back_addr":"172.21.15.28:6807/2964323435","heartbeat_front_addr":"172.21.15.28:6805/2964323435","state":["exists","up"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6809","nonce":23768570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6811","nonce":23768570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6815","nonce":23768570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6813","nonce":23768570}]},"public_addr":"172.21.15.28:6809/23768570","cluster_addr":"172.21.15.28:6811/23768570","heartbeat_back_addr":"172.21.15.28:6815/23768570","heartbeat_front_addr":"172.21.15.28:6813/23768570","state":["exists","up"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6817","nonce":886887303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6819","nonce":886887303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6823","nonce":886887303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6821","nonce":886887303}]},"public_addr":"172.21.15.28:6817/886887303","cluster_addr":"172.21.15.28:6819/886887303","heartbeat_back_addr":"172.21.15.28:6823/886887303","heartbeat_front_addr":"172.21.15.28:6821/886887303","state":["exists","up"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6825","nonce":2616036674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6827","nonce":2616036674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6831","nonce":2616036674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6829","nonce":2616036674}]},"public_addr":"172.21.15.28:6825/2616036674","cluster_addr":"172.21.15.28:6827/2616036674","heartbeat_back_addr":"172.21.15.28:6831/2616036674","heartbeat_front_addr":"172.21.15.28:6829/2616036674","state":["exists","up"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6801","nonce":2817102450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6803","nonce":2817102450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6807","nonce":2817102450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6805","nonce":2817102450}]},"public_addr":"172.21.15.191:6801/2817102450","cluster_addr":"172.21.15.191:6803/2817102450","heartbeat_back_addr":"172.21.15.191:6807/2817102450","heartbeat_front_addr":"172.21.15.191:6805/2817102450","state":["exists","up"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6817","nonce":2503027971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6819","nonce":2503027971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6823","nonce":2503027971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6821","nonce":2503027971}]},"public_addr":"172.21.15.191:6817/2503027971","cluster_addr":"172.21.15.191:6819/2503027971","heartbeat_back_addr":"172.21.15.191:6823/2503027971","heartbeat_front_addr":"172.21.15.191:6821/2503027971","state":["exists","up"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6809","nonce":3006595088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6811","nonce":3006595088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6815","nonce":3006595088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6813","nonce":3006595088}]},"public_addr":"172.21.15.191:6809/3006595088","cluster_addr":"172.21.15.191:6811/3006595088","heartbeat_back_addr":"172.21.15.191:6815/3006595088","heartbeat_front_addr":"172.21.15.191:6813/3006595088","state":["exists","up"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6825","nonce":1477327720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6827","nonce":1477327720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6831","nonce":1477327720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6829","nonce":1477327720}]},"public_addr":"172.21.15.191:6825/1477327720","cluster_addr":"172.21.15.191:6827/1477327720","heartbeat_back_addr":"172.21.15.191:6831/1477327720","heartbeat_front_addr":"172.21.15.191:6829/1477327720","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.917250+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.954351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.017329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.046221+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.707467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.726947+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.782321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.764850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:44:00.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.773+0000 7ff6d7c00640 1 -- 172.21.15.28:0/1336083969 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff6ac06ac30 msgr2=0x7ff6ac06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.773+0000 7ff6d7c00640 1 --2- 172.21.15.28:0/1336083969 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff6ac06ac30 0x7ff6ac06d0f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0000f60 tx=0x7ff6c0009040 comp rx=0 tx=0).stop 2024-09-13T16:44:00.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.773+0000 7ff6d7c00640 1 -- 172.21.15.28:0/1336083969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d00c5bd0 msgr2=0x7ff6d00bd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.773+0000 7ff6d7c00640 1 --2- 172.21.15.28:0/1336083969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d00c5bd0 0x7ff6d00bd850 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c7b0 tx=0x7ff6c800cc70 comp rx=0 tx=0).stop 2024-09-13T16:44:00.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.773+0000 7ff6d7c00640 1 -- 172.21.15.28:0/1336083969 shutdown_connections 2024-09-13T16:44:00.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.773+0000 7ff6d7c00640 1 --2- 172.21.15.28:0/1336083969 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff6ac06ac30 0x7ff6ac06d0f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.773+0000 7ff6d7c00640 1 --2- 172.21.15.28:0/1336083969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6d00c5bd0 0x7ff6d00bd850 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.773+0000 7ff6d7c00640 1 --2- 172.21.15.28:0/1336083969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6d00c3670 0x7ff6d01ee6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.773+0000 7ff6d7c00640 1 --2- 172.21.15.28:0/1336083969 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6d00c1ea0 0x7ff6d0144160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.773+0000 7ff6d7c00640 1 -- 172.21.15.28:0/1336083969 >> 172.21.15.28:0/1336083969 conn(0x7ff6d0066040 msgr2=0x7ff6d0070fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.773+0000 7ff6d7c00640 1 -- 172.21.15.28:0/1336083969 shutdown_connections 2024-09-13T16:44:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.773+0000 7ff6d7c00640 1 -- 172.21.15.28:0/1336083969 wait complete. 2024-09-13T16:44:00.790 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T16:44:00.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:44:00.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.941+0000 7f8575000640 1 Processor -- start 2024-09-13T16:44:00.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.942+0000 7f8575000640 1 -- start start 2024-09-13T16:44:00.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.943+0000 7f8575000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f85701593e0 0x7f85700c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:00.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.943+0000 7f8575000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85701598d0 con 0x7f857015cc20 2024-09-13T16:44:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.943+0000 7f8575000640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85700aede0 con 0x7f8570152910 2024-09-13T16:44:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.943+0000 7f8575000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85700af0a0 con 0x7f85701593e0 2024-09-13T16:44:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.943+0000 7f856fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f85701593e0 0x7f85700c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.943+0000 7f8566a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f857015cc20 0x7f857006fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59466/0 (socket says 172.21.15.28:59466) 2024-09-13T16:44:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.943+0000 7f8566a00640 1 -- 172.21.15.28:0/2717710852 learned_addr learned my addr 172.21.15.28:0/2717710852 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.943+0000 7f856fe00640 1 -- 172.21.15.28:0/2717710852 >> v1:172.21.15.191:6789/0 conn(0x7f8570152910 legacy=0x7f85700c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.943+0000 7f856fe00640 1 -- 172.21.15.28:0/2717710852 >> v1:172.21.15.28:6789/0 conn(0x7f857015cc20 legacy=0x7f857006fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.943+0000 7f856fe00640 1 -- 172.21.15.28:0/2717710852 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857015c140 con 0x7f85701593e0 2024-09-13T16:44:00.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.944+0000 7f856fe00640 1 --2- 172.21.15.28:0/2717710852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85701593e0 0x7f85700c6ec0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8568009d90 tx=0x7f856802a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aefc7d374912593a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:00.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.945+0000 7f856ea00640 1 -- 172.21.15.28:0/2717710852 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8568007740 con 0x7f85701593e0 2024-09-13T16:44:00.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.945+0000 7f856ea00640 1 -- 172.21.15.28:0/2717710852 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85680078a0 con 0x7f85701593e0 2024-09-13T16:44:00.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.945+0000 7f856ea00640 1 -- 172.21.15.28:0/2717710852 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8568007c30 con 0x7f85701593e0 2024-09-13T16:44:00.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.945+0000 7f8575000640 1 -- 172.21.15.28:0/2717710852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85701593e0 msgr2=0x7f85700c6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.945+0000 7f8575000640 1 --2- 172.21.15.28:0/2717710852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85701593e0 0x7f85700c6ec0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8568009d90 tx=0x7f856802a780 comp rx=0 tx=0).stop 2024-09-13T16:44:00.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.945+0000 7f8575000640 1 -- 172.21.15.28:0/2717710852 shutdown_connections 2024-09-13T16:44:00.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.945+0000 7f8575000640 1 --2- 172.21.15.28:0/2717710852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85701593e0 0x7f85700c6ec0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.945+0000 7f8575000640 1 -- 172.21.15.28:0/2717710852 >> 172.21.15.28:0/2717710852 conn(0x7f857006e3d0 msgr2=0x7f8570069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:00.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.945+0000 7f8575000640 1 -- 172.21.15.28:0/2717710852 shutdown_connections 2024-09-13T16:44:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.945+0000 7f8575000640 1 -- 172.21.15.28:0/2717710852 wait complete. 2024-09-13T16:44:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.947+0000 7f8575000640 1 Processor -- start 2024-09-13T16:44:00.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.947+0000 7f8575000640 1 -- start start 2024-09-13T16:44:00.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.948+0000 7f8575000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8570152910 0x7f85701eee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:00.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.948+0000 7f8575000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85701593e0 0x7f85700c3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:00.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.948+0000 7f8575000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f857015cc20 0x7f857015b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:00.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.948+0000 7f8575000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85701ef3d0 con 0x7f857015cc20 2024-09-13T16:44:00.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.948+0000 7f8575000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85700c5160 con 0x7f85701593e0 2024-09-13T16:44:00.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.948+0000 7f8575000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85700c62c0 con 0x7f8570152910 2024-09-13T16:44:00.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.948+0000 7f856fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f857015cc20 0x7f857015b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:00.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.948+0000 7f856fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f857015cc20 0x7f857015b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41350/0 (socket says 172.21.15.28:41350) 2024-09-13T16:44:00.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.948+0000 7f856fe00640 1 -- 172.21.15.28:0/3632645320 learned_addr learned my addr 172.21.15.28:0/3632645320 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:00.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.948+0000 7f856f400640 1 --2- 172.21.15.28:0/3632645320 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8570152910 0x7f85701eee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:00.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.948+0000 7f8566a00640 1 --2- 172.21.15.28:0/3632645320 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85701593e0 0x7f85700c3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:00.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.948+0000 7f856fe00640 1 -- 172.21.15.28:0/3632645320 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8570152910 msgr2=0x7f85701eee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.948+0000 7f856fe00640 1 --2- 172.21.15.28:0/3632645320 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8570152910 0x7f85701eee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.948+0000 7f856fe00640 1 -- 172.21.15.28:0/3632645320 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85701593e0 msgr2=0x7f85700c3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.948+0000 7f856fe00640 1 --2- 172.21.15.28:0/3632645320 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85701593e0 0x7f85700c3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.948+0000 7f856fe00640 1 -- 172.21.15.28:0/3632645320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8568009a70 con 0x7f857015cc20 2024-09-13T16:44:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.949+0000 7f8566a00640 1 --2- 172.21.15.28:0/3632645320 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85701593e0 0x7f85700c3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.949+0000 7f856fe00640 1 --2- 172.21.15.28:0/3632645320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f857015cc20 0x7f857015b470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8568009eb0 tx=0x7f8568004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.949+0000 7f8567e00640 1 -- 172.21.15.28:0/3632645320 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8568032c00 con 0x7f857015cc20 2024-09-13T16:44:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.949+0000 7f8567e00640 1 -- 172.21.15.28:0/3632645320 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8568032d60 con 0x7f857015cc20 2024-09-13T16:44:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.949+0000 7f8567e00640 1 -- 172.21.15.28:0/3632645320 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856803c650 con 0x7f857015cc20 2024-09-13T16:44:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.949+0000 7f8575000640 1 -- 172.21.15.28:0/3632645320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85700c4010 con 0x7f857015cc20 2024-09-13T16:44:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.949+0000 7f8575000640 1 -- 172.21.15.28:0/3632645320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85700b2840 con 0x7f857015cc20 2024-09-13T16:44:00.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.950+0000 7f8575000640 1 -- 172.21.15.28:0/3632645320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85700c74d0 con 0x7f857015cc20 2024-09-13T16:44:00.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.951+0000 7f8567e00640 1 -- 172.21.15.28:0/3632645320 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8568039050 con 0x7f857015cc20 2024-09-13T16:44:00.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.952+0000 7f8567e00640 1 --2- 172.21.15.28:0/3632645320 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f854c06ac00 0x7f854c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:00.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.952+0000 7f8567e00640 1 -- 172.21.15.28:0/3632645320 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85680aa580 con 0x7f857015cc20 2024-09-13T16:44:00.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.952+0000 7f856f400640 1 --2- 172.21.15.28:0/3632645320 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f854c06ac00 0x7f854c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:00.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.953+0000 7f856f400640 1 --2- 172.21.15.28:0/3632645320 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f854c06ac00 0x7f854c06d0c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8560000f60 tx=0x7f8560009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:00.956+0000 7f8567e00640 1 -- 172.21.15.28:0/3632645320 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f85680757a0 con 0x7f857015cc20 2024-09-13T16:44:01.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.131+0000 7f8575000640 1 -- 172.21.15.28:0/3632645320 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8570152cf0 con 0x7f854c06ac00 2024-09-13T16:44:01.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.139+0000 7f8567e00640 1 -- 172.21.15.28:0/3632645320 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264611 (secure 0 0 0) 0x7f8560001d10 con 0x7f854c06ac00 2024-09-13T16:44:01.143 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T16:44:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.142+0000 7f8575000640 1 -- 172.21.15.28:0/3632645320 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f854c06ac00 msgr2=0x7f854c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.142+0000 7f8575000640 1 --2- 172.21.15.28:0/3632645320 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f854c06ac00 0x7f854c06d0c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8560000f60 tx=0x7f8560009040 comp rx=0 tx=0).stop 2024-09-13T16:44:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.142+0000 7f8575000640 1 -- 172.21.15.28:0/3632645320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f857015cc20 msgr2=0x7f857015b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.142+0000 7f8575000640 1 --2- 172.21.15.28:0/3632645320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f857015cc20 0x7f857015b470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8568009eb0 tx=0x7f8568004a40 comp rx=0 tx=0).stop 2024-09-13T16:44:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.142+0000 7f8575000640 1 -- 172.21.15.28:0/3632645320 shutdown_connections 2024-09-13T16:44:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.142+0000 7f8575000640 1 --2- 172.21.15.28:0/3632645320 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f854c06ac00 0x7f854c06d0c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.143+0000 7f8575000640 1 --2- 172.21.15.28:0/3632645320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f857015cc20 0x7f857015b470 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.143+0000 7f8575000640 1 --2- 172.21.15.28:0/3632645320 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85701593e0 0x7f85700c3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.143+0000 7f8575000640 1 --2- 172.21.15.28:0/3632645320 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8570152910 0x7f85701eee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.143+0000 7f8575000640 1 -- 172.21.15.28:0/3632645320 >> 172.21.15.28:0/3632645320 conn(0x7f857006e3d0 msgr2=0x7f857006fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:01.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.143+0000 7f8575000640 1 -- 172.21.15.28:0/3632645320 shutdown_connections 2024-09-13T16:44:01.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.143+0000 7f8575000640 1 -- 172.21.15.28:0/3632645320 wait complete. 2024-09-13T16:44:01.160 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-13T16:43:59.591349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221988,"kb_used_data":1680,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754752732,"statfs":{"total":773094113280,"available":772866797568,"internally_reserved":0,"allocated":1720320,"data_stored":1726298,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12743,"internal_metadata":225431097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.058679"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178474+0000","last_change":"2024-09-13T16:43:54.987241+0000","last_active":"2024-09-13T16:43:59.178474+0000","last_peered":"2024-09-13T16:43:59.178474+0000","last_clean":"2024-09-13T16:43:59.178474+0000","last_became_active":"2024-09-13T16:43:54.986988+0000","last_became_peered":"2024-09-13T16:43:54.986988+0000","last_unstale":"2024-09-13T16:43:59.178474+0000","last_undegraded":"2024-09-13T16:43:59.178474+0000","last_fullsized":"2024-09-13T16:43:59.178474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975685+0000","last_change":"2024-09-13T16:43:55.001141+0000","last_active":"2024-09-13T16:43:55.975685+0000","last_peered":"2024-09-13T16:43:55.975685+0000","last_clean":"2024-09-13T16:43:55.975685+0000","last_became_active":"2024-09-13T16:43:55.000757+0000","last_became_peered":"2024-09-13T16:43:55.000757+0000","last_unstale":"2024-09-13T16:43:55.975685+0000","last_undegraded":"2024-09-13T16:43:55.975685+0000","last_fullsized":"2024-09-13T16:43:55.975685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872108+0000","last_change":"2024-09-13T16:43:54.986925+0000","last_active":"2024-09-13T16:43:56.872108+0000","last_peered":"2024-09-13T16:43:56.872108+0000","last_clean":"2024-09-13T16:43:56.872108+0000","last_became_active":"2024-09-13T16:43:54.986198+0000","last_became_peered":"2024-09-13T16:43:54.986198+0000","last_unstale":"2024-09-13T16:43:56.872108+0000","last_undegraded":"2024-09-13T16:43:56.872108+0000","last_fullsized":"2024-09-13T16:43:56.872108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:03.127550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.995272+0000","last_change":"2024-09-13T16:43:54.992249+0000","last_active":"2024-09-13T16:43:54.995272+0000","last_peered":"2024-09-13T16:43:54.995272+0000","last_clean":"2024-09-13T16:43:54.995272+0000","last_became_active":"2024-09-13T16:43:54.991804+0000","last_became_peered":"2024-09-13T16:43:54.991804+0000","last_unstale":"2024-09-13T16:43:54.995272+0000","last_undegraded":"2024-09-13T16:43:54.995272+0000","last_fullsized":"2024-09-13T16:43:54.995272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178163+0000","last_change":"2024-09-13T16:43:54.987564+0000","last_active":"2024-09-13T16:43:59.178163+0000","last_peered":"2024-09-13T16:43:59.178163+0000","last_clean":"2024-09-13T16:43:59.178163+0000","last_became_active":"2024-09-13T16:43:54.987328+0000","last_became_peered":"2024-09-13T16:43:54.987328+0000","last_unstale":"2024-09-13T16:43:59.178163+0000","last_undegraded":"2024-09-13T16:43:59.178163+0000","last_fullsized":"2024-09-13T16:43:59.178163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:36.006730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.005629+0000","last_change":"2024-09-13T16:43:55.000019+0000","last_active":"2024-09-13T16:43:55.005629+0000","last_peered":"2024-09-13T16:43:55.005629+0000","last_clean":"2024-09-13T16:43:55.005629+0000","last_became_active":"2024-09-13T16:43:54.999752+0000","last_became_peered":"2024-09-13T16:43:54.999752+0000","last_unstale":"2024-09-13T16:43:55.005629+0000","last_undegraded":"2024-09-13T16:43:55.005629+0000","last_fullsized":"2024-09-13T16:43:55.005629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.177985+0000","last_change":"2024-09-13T16:43:54.999533+0000","last_active":"2024-09-13T16:43:59.177985+0000","last_peered":"2024-09-13T16:43:59.177985+0000","last_clean":"2024-09-13T16:43:59.177985+0000","last_became_active":"2024-09-13T16:43:54.999350+0000","last_became_peered":"2024-09-13T16:43:54.999350+0000","last_unstale":"2024-09-13T16:43:59.177985+0000","last_undegraded":"2024-09-13T16:43:59.177985+0000","last_fullsized":"2024-09-13T16:43:59.177985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:29.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508531+0000","last_change":"2024-09-13T16:43:54.998526+0000","last_active":"2024-09-13T16:43:56.508531+0000","last_peered":"2024-09-13T16:43:56.508531+0000","last_clean":"2024-09-13T16:43:56.508531+0000","last_became_active":"2024-09-13T16:43:54.996309+0000","last_became_peered":"2024-09-13T16:43:54.996309+0000","last_unstale":"2024-09-13T16:43:56.508531+0000","last_undegraded":"2024-09-13T16:43:56.508531+0000","last_fullsized":"2024-09-13T16:43:56.508531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178473+0000","last_change":"2024-09-13T16:43:54.993786+0000","last_active":"2024-09-13T16:43:59.178473+0000","last_peered":"2024-09-13T16:43:59.178473+0000","last_clean":"2024-09-13T16:43:59.178473+0000","last_became_active":"2024-09-13T16:43:54.993475+0000","last_became_peered":"2024-09-13T16:43:54.993475+0000","last_unstale":"2024-09-13T16:43:59.178473+0000","last_undegraded":"2024-09-13T16:43:59.178473+0000","last_fullsized":"2024-09-13T16:43:59.178473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260988+0000","last_change":"2024-09-13T16:43:54.986272+0000","last_active":"2024-09-13T16:43:56.260988+0000","last_peered":"2024-09-13T16:43:56.260988+0000","last_clean":"2024-09-13T16:43:56.260988+0000","last_became_active":"2024-09-13T16:43:54.985979+0000","last_became_peered":"2024-09-13T16:43:54.985979+0000","last_unstale":"2024-09-13T16:43:56.260988+0000","last_undegraded":"2024-09-13T16:43:56.260988+0000","last_fullsized":"2024-09-13T16:43:56.260988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260766+0000","last_change":"2024-09-13T16:43:54.986207+0000","last_active":"2024-09-13T16:43:56.260766+0000","last_peered":"2024-09-13T16:43:56.260766+0000","last_clean":"2024-09-13T16:43:56.260766+0000","last_became_active":"2024-09-13T16:43:54.985948+0000","last_became_peered":"2024-09-13T16:43:54.985948+0000","last_unstale":"2024-09-13T16:43:56.260766+0000","last_undegraded":"2024-09-13T16:43:56.260766+0000","last_fullsized":"2024-09-13T16:43:56.260766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935591+0000","last_change":"2024-09-13T16:43:54.986636+0000","last_active":"2024-09-13T16:43:56.935591+0000","last_peered":"2024-09-13T16:43:56.935591+0000","last_clean":"2024-09-13T16:43:56.935591+0000","last_became_active":"2024-09-13T16:43:54.986161+0000","last_became_peered":"2024-09-13T16:43:54.986161+0000","last_unstale":"2024-09-13T16:43:56.935591+0000","last_undegraded":"2024-09-13T16:43:56.935591+0000","last_fullsized":"2024-09-13T16:43:56.935591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:09.340330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508720+0000","last_change":"2024-09-13T16:43:54.997783+0000","last_active":"2024-09-13T16:43:56.508720+0000","last_peered":"2024-09-13T16:43:56.508720+0000","last_clean":"2024-09-13T16:43:56.508720+0000","last_became_active":"2024-09-13T16:43:54.997162+0000","last_became_peered":"2024-09-13T16:43:54.997162+0000","last_unstale":"2024-09-13T16:43:56.508720+0000","last_undegraded":"2024-09-13T16:43:56.508720+0000","last_fullsized":"2024-09-13T16:43:56.508720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872216+0000","last_change":"2024-09-13T16:43:54.995262+0000","last_active":"2024-09-13T16:43:56.872216+0000","last_peered":"2024-09-13T16:43:56.872216+0000","last_clean":"2024-09-13T16:43:56.872216+0000","last_became_active":"2024-09-13T16:43:54.994891+0000","last_became_peered":"2024-09-13T16:43:54.994891+0000","last_unstale":"2024-09-13T16:43:56.872216+0000","last_undegraded":"2024-09-13T16:43:56.872216+0000","last_fullsized":"2024-09-13T16:43:56.872216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975286+0000","last_change":"2024-09-13T16:43:54.981534+0000","last_active":"2024-09-13T16:43:55.975286+0000","last_peered":"2024-09-13T16:43:55.975286+0000","last_clean":"2024-09-13T16:43:55.975286+0000","last_became_active":"2024-09-13T16:43:54.981191+0000","last_became_peered":"2024-09-13T16:43:54.981191+0000","last_unstale":"2024-09-13T16:43:55.975286+0000","last_undegraded":"2024-09-13T16:43:55.975286+0000","last_fullsized":"2024-09-13T16:43:55.975286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.997667+0000","last_change":"2024-09-13T16:43:54.994157+0000","last_active":"2024-09-13T16:43:54.997667+0000","last_peered":"2024-09-13T16:43:54.997667+0000","last_clean":"2024-09-13T16:43:54.997667+0000","last_became_active":"2024-09-13T16:43:54.993915+0000","last_became_peered":"2024-09-13T16:43:54.993915+0000","last_unstale":"2024-09-13T16:43:54.997667+0000","last_undegraded":"2024-09-13T16:43:54.997667+0000","last_fullsized":"2024-09-13T16:43:54.997667+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.995002+0000","last_change":"2024-09-13T16:43:54.987328+0000","last_active":"2024-09-13T16:43:54.995002+0000","last_peered":"2024-09-13T16:43:54.995002+0000","last_clean":"2024-09-13T16:43:54.995002+0000","last_became_active":"2024-09-13T16:43:54.987048+0000","last_became_peered":"2024-09-13T16:43:54.987048+0000","last_unstale":"2024-09-13T16:43:54.995002+0000","last_undegraded":"2024-09-13T16:43:54.995002+0000","last_fullsized":"2024-09-13T16:43:54.995002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.979840+0000","last_change":"2024-09-13T16:43:54.985628+0000","last_active":"2024-09-13T16:43:55.979840+0000","last_peered":"2024-09-13T16:43:55.979840+0000","last_clean":"2024-09-13T16:43:55.979840+0000","last_became_active":"2024-09-13T16:43:54.985378+0000","last_became_peered":"2024-09-13T16:43:54.985378+0000","last_unstale":"2024-09-13T16:43:55.979840+0000","last_undegraded":"2024-09-13T16:43:55.979840+0000","last_fullsized":"2024-09-13T16:43:55.979840+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178469+0000","last_change":"2024-09-13T16:43:54.996712+0000","last_active":"2024-09-13T16:43:59.178469+0000","last_peered":"2024-09-13T16:43:59.178469+0000","last_clean":"2024-09-13T16:43:59.178469+0000","last_became_active":"2024-09-13T16:43:54.996523+0000","last_became_peered":"2024-09-13T16:43:54.996523+0000","last_unstale":"2024-09-13T16:43:59.178469+0000","last_undegraded":"2024-09-13T16:43:59.178469+0000","last_fullsized":"2024-09-13T16:43:59.178469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260867+0000","last_change":"2024-09-13T16:43:54.984879+0000","last_active":"2024-09-13T16:43:56.260867+0000","last_peered":"2024-09-13T16:43:56.260867+0000","last_clean":"2024-09-13T16:43:56.260867+0000","last_became_active":"2024-09-13T16:43:54.984595+0000","last_became_peered":"2024-09-13T16:43:54.984595+0000","last_unstale":"2024-09-13T16:43:56.260867+0000","last_undegraded":"2024-09-13T16:43:56.260867+0000","last_fullsized":"2024-09-13T16:43:56.260867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.871914+0000","last_change":"2024-09-13T16:43:54.996457+0000","last_active":"2024-09-13T16:43:56.871914+0000","last_peered":"2024-09-13T16:43:56.871914+0000","last_clean":"2024-09-13T16:43:56.871914+0000","last_became_active":"2024-09-13T16:43:54.996275+0000","last_became_peered":"2024-09-13T16:43:54.996275+0000","last_unstale":"2024-09-13T16:43:56.871914+0000","last_undegraded":"2024-09-13T16:43:56.871914+0000","last_fullsized":"2024-09-13T16:43:56.871914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.993287+0000","last_change":"2024-09-13T16:43:54.985368+0000","last_active":"2024-09-13T16:43:54.993287+0000","last_peered":"2024-09-13T16:43:54.993287+0000","last_clean":"2024-09-13T16:43:54.993287+0000","last_became_active":"2024-09-13T16:43:54.985130+0000","last_became_peered":"2024-09-13T16:43:54.985130+0000","last_unstale":"2024-09-13T16:43:54.993287+0000","last_undegraded":"2024-09-13T16:43:54.993287+0000","last_fullsized":"2024-09-13T16:43:54.993287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508531+0000","last_change":"2024-09-13T16:43:54.991862+0000","last_active":"2024-09-13T16:43:56.508531+0000","last_peered":"2024-09-13T16:43:56.508531+0000","last_clean":"2024-09-13T16:43:56.508531+0000","last_became_active":"2024-09-13T16:43:54.991554+0000","last_became_peered":"2024-09-13T16:43:54.991554+0000","last_unstale":"2024-09-13T16:43:56.508531+0000","last_undegraded":"2024-09-13T16:43:56.508531+0000","last_fullsized":"2024-09-13T16:43:56.508531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.995238+0000","last_change":"2024-09-13T16:43:54.992241+0000","last_active":"2024-09-13T16:43:54.995238+0000","last_peered":"2024-09-13T16:43:54.995238+0000","last_clean":"2024-09-13T16:43:54.995238+0000","last_became_active":"2024-09-13T16:43:54.991687+0000","last_became_peered":"2024-09-13T16:43:54.991687+0000","last_unstale":"2024-09-13T16:43:54.995238+0000","last_undegraded":"2024-09-13T16:43:54.995238+0000","last_fullsized":"2024-09-13T16:43:54.995238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.509024+0000","last_change":"2024-09-13T16:43:54.991366+0000","last_active":"2024-09-13T16:43:56.509024+0000","last_peered":"2024-09-13T16:43:56.509024+0000","last_clean":"2024-09-13T16:43:56.509024+0000","last_became_active":"2024-09-13T16:43:54.991127+0000","last_became_peered":"2024-09-13T16:43:54.991127+0000","last_unstale":"2024-09-13T16:43:56.509024+0000","last_undegraded":"2024-09-13T16:43:56.509024+0000","last_fullsized":"2024-09-13T16:43:56.509024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.261006+0000","last_change":"2024-09-13T16:43:54.985615+0000","last_active":"2024-09-13T16:43:56.261006+0000","last_peered":"2024-09-13T16:43:56.261006+0000","last_clean":"2024-09-13T16:43:56.261006+0000","last_became_active":"2024-09-13T16:43:54.985299+0000","last_became_peered":"2024-09-13T16:43:54.985299+0000","last_unstale":"2024-09-13T16:43:56.261006+0000","last_undegraded":"2024-09-13T16:43:56.261006+0000","last_fullsized":"2024-09-13T16:43:56.261006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975382+0000","last_change":"2024-09-13T16:43:54.993325+0000","last_active":"2024-09-13T16:43:55.975382+0000","last_peered":"2024-09-13T16:43:55.975382+0000","last_clean":"2024-09-13T16:43:55.975382+0000","last_became_active":"2024-09-13T16:43:54.993009+0000","last_became_peered":"2024-09-13T16:43:54.993009+0000","last_unstale":"2024-09-13T16:43:55.975382+0000","last_undegraded":"2024-09-13T16:43:55.975382+0000","last_fullsized":"2024-09-13T16:43:55.975382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508820+0000","last_change":"2024-09-13T16:43:54.998229+0000","last_active":"2024-09-13T16:43:56.508820+0000","last_peered":"2024-09-13T16:43:56.508820+0000","last_clean":"2024-09-13T16:43:56.508820+0000","last_became_active":"2024-09-13T16:43:54.995774+0000","last_became_peered":"2024-09-13T16:43:54.995774+0000","last_unstale":"2024-09-13T16:43:56.508820+0000","last_undegraded":"2024-09-13T16:43:56.508820+0000","last_fullsized":"2024-09-13T16:43:56.508820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872235+0000","last_change":"2024-09-13T16:43:55.002126+0000","last_active":"2024-09-13T16:43:56.872235+0000","last_peered":"2024-09-13T16:43:56.872235+0000","last_clean":"2024-09-13T16:43:56.872235+0000","last_became_active":"2024-09-13T16:43:55.001713+0000","last_became_peered":"2024-09-13T16:43:55.001713+0000","last_unstale":"2024-09-13T16:43:56.872235+0000","last_undegraded":"2024-09-13T16:43:56.872235+0000","last_fullsized":"2024-09-13T16:43:56.872235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.995327+0000","last_change":"2024-09-13T16:43:54.992745+0000","last_active":"2024-09-13T16:43:54.995327+0000","last_peered":"2024-09-13T16:43:54.995327+0000","last_clean":"2024-09-13T16:43:54.995327+0000","last_became_active":"2024-09-13T16:43:54.992508+0000","last_became_peered":"2024-09-13T16:43:54.992508+0000","last_unstale":"2024-09-13T16:43:54.995327+0000","last_undegraded":"2024-09-13T16:43:54.995327+0000","last_fullsized":"2024-09-13T16:43:54.995327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:23.913470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.979793+0000","last_change":"2024-09-13T16:43:54.990631+0000","last_active":"2024-09-13T16:43:55.979793+0000","last_peered":"2024-09-13T16:43:55.979793+0000","last_clean":"2024-09-13T16:43:55.979793+0000","last_became_active":"2024-09-13T16:43:54.986277+0000","last_became_peered":"2024-09-13T16:43:54.986277+0000","last_unstale":"2024-09-13T16:43:55.979793+0000","last_undegraded":"2024-09-13T16:43:55.979793+0000","last_fullsized":"2024-09-13T16:43:55.979793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260674+0000","last_change":"2024-09-13T16:43:55.000059+0000","last_active":"2024-09-13T16:43:56.260674+0000","last_peered":"2024-09-13T16:43:56.260674+0000","last_clean":"2024-09-13T16:43:56.260674+0000","last_became_active":"2024-09-13T16:43:54.999778+0000","last_became_peered":"2024-09-13T16:43:54.999778+0000","last_unstale":"2024-09-13T16:43:56.260674+0000","last_undegraded":"2024-09-13T16:43:56.260674+0000","last_fullsized":"2024-09-13T16:43:56.260674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935551+0000","last_change":"2024-09-13T16:43:54.988375+0000","last_active":"2024-09-13T16:43:56.935551+0000","last_peered":"2024-09-13T16:43:56.935551+0000","last_clean":"2024-09-13T16:43:56.935551+0000","last_became_active":"2024-09-13T16:43:54.988131+0000","last_became_peered":"2024-09-13T16:43:54.988131+0000","last_unstale":"2024-09-13T16:43:56.935551+0000","last_undegraded":"2024-09-13T16:43:56.935551+0000","last_fullsized":"2024-09-13T16:43:56.935551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.635755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.985641+0000","last_change":"2024-09-13T16:43:54.555042+0000","last_active":"2024-09-13T16:43:54.985641+0000","last_peered":"2024-09-13T16:43:54.985641+0000","last_clean":"2024-09-13T16:43:54.985641+0000","last_became_active":"2024-09-13T16:43:54.554697+0000","last_became_peered":"2024-09-13T16:43:54.554697+0000","last_unstale":"2024-09-13T16:43:54.985641+0000","last_undegraded":"2024-09-13T16:43:54.985641+0000","last_fullsized":"2024-09-13T16:43:54.985641+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872075+0000","last_change":"2024-09-13T16:43:54.998755+0000","last_active":"2024-09-13T16:43:56.872075+0000","last_peered":"2024-09-13T16:43:56.872075+0000","last_clean":"2024-09-13T16:43:56.872075+0000","last_became_active":"2024-09-13T16:43:54.998399+0000","last_became_peered":"2024-09-13T16:43:54.998399+0000","last_unstale":"2024-09-13T16:43:56.872075+0000","last_undegraded":"2024-09-13T16:43:56.872075+0000","last_fullsized":"2024-09-13T16:43:56.872075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.978215+0000","last_change":"2024-09-13T16:43:54.553250+0000","last_active":"2024-09-13T16:43:54.978215+0000","last_peered":"2024-09-13T16:43:54.978215+0000","last_clean":"2024-09-13T16:43:54.978215+0000","last_became_active":"2024-09-13T16:43:54.552977+0000","last_became_peered":"2024-09-13T16:43:54.552977+0000","last_unstale":"2024-09-13T16:43:54.978215+0000","last_undegraded":"2024-09-13T16:43:54.978215+0000","last_fullsized":"2024-09-13T16:43:54.978215+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.984357+0000","last_change":"2024-09-13T16:43:54.558065+0000","last_active":"2024-09-13T16:43:54.984357+0000","last_peered":"2024-09-13T16:43:54.984357+0000","last_clean":"2024-09-13T16:43:54.984357+0000","last_became_active":"2024-09-13T16:43:54.557722+0000","last_became_peered":"2024-09-13T16:43:54.557722+0000","last_unstale":"2024-09-13T16:43:54.984357+0000","last_undegraded":"2024-09-13T16:43:54.984357+0000","last_fullsized":"2024-09-13T16:43:54.984357+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508519+0000","last_change":"2024-09-13T16:43:55.002041+0000","last_active":"2024-09-13T16:43:56.508519+0000","last_peered":"2024-09-13T16:43:56.508519+0000","last_clean":"2024-09-13T16:43:56.508519+0000","last_became_active":"2024-09-13T16:43:55.001512+0000","last_became_peered":"2024-09-13T16:43:55.001512+0000","last_unstale":"2024-09-13T16:43:56.508519+0000","last_undegraded":"2024-09-13T16:43:56.508519+0000","last_fullsized":"2024-09-13T16:43:56.508519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.975255+0000","last_change":"2024-09-13T16:43:54.550389+0000","last_active":"2024-09-13T16:43:54.975255+0000","last_peered":"2024-09-13T16:43:54.975255+0000","last_clean":"2024-09-13T16:43:54.975255+0000","last_became_active":"2024-09-13T16:43:54.550154+0000","last_became_peered":"2024-09-13T16:43:54.550154+0000","last_unstale":"2024-09-13T16:43:54.975255+0000","last_undegraded":"2024-09-13T16:43:54.975255+0000","last_fullsized":"2024-09-13T16:43:54.975255+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975641+0000","last_change":"2024-09-13T16:43:54.993377+0000","last_active":"2024-09-13T16:43:55.975641+0000","last_peered":"2024-09-13T16:43:55.975641+0000","last_clean":"2024-09-13T16:43:55.975641+0000","last_became_active":"2024-09-13T16:43:54.993036+0000","last_became_peered":"2024-09-13T16:43:54.993036+0000","last_unstale":"2024-09-13T16:43:55.975641+0000","last_undegraded":"2024-09-13T16:43:55.975641+0000","last_fullsized":"2024-09-13T16:43:55.975641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.978710+0000","last_change":"2024-09-13T16:43:54.555528+0000","last_active":"2024-09-13T16:43:54.978710+0000","last_peered":"2024-09-13T16:43:54.978710+0000","last_clean":"2024-09-13T16:43:54.978710+0000","last_became_active":"2024-09-13T16:43:54.555267+0000","last_became_peered":"2024-09-13T16:43:54.555267+0000","last_unstale":"2024-09-13T16:43:54.978710+0000","last_undegraded":"2024-09-13T16:43:54.978710+0000","last_fullsized":"2024-09-13T16:43:54.978710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.989737+0000","last_change":"2024-09-13T16:43:54.981706+0000","last_active":"2024-09-13T16:43:54.989737+0000","last_peered":"2024-09-13T16:43:54.989737+0000","last_clean":"2024-09-13T16:43:54.989737+0000","last_became_active":"2024-09-13T16:43:54.981429+0000","last_became_peered":"2024-09-13T16:43:54.981429+0000","last_unstale":"2024-09-13T16:43:54.989737+0000","last_undegraded":"2024-09-13T16:43:54.989737+0000","last_fullsized":"2024-09-13T16:43:54.989737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.975723+0000","last_change":"2024-09-13T16:43:54.554593+0000","last_active":"2024-09-13T16:43:54.975723+0000","last_peered":"2024-09-13T16:43:54.975723+0000","last_clean":"2024-09-13T16:43:54.975723+0000","last_became_active":"2024-09-13T16:43:54.554307+0000","last_became_peered":"2024-09-13T16:43:54.554307+0000","last_unstale":"2024-09-13T16:43:54.975723+0000","last_undegraded":"2024-09-13T16:43:54.975723+0000","last_fullsized":"2024-09-13T16:43:54.975723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872139+0000","last_change":"2024-09-13T16:43:54.993485+0000","last_active":"2024-09-13T16:43:56.872139+0000","last_peered":"2024-09-13T16:43:56.872139+0000","last_clean":"2024-09-13T16:43:56.872139+0000","last_became_active":"2024-09-13T16:43:54.993226+0000","last_became_peered":"2024-09-13T16:43:54.993226+0000","last_unstale":"2024-09-13T16:43:56.872139+0000","last_undegraded":"2024-09-13T16:43:56.872139+0000","last_fullsized":"2024-09-13T16:43:56.872139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.975089+0000","last_change":"2024-09-13T16:43:54.562976+0000","last_active":"2024-09-13T16:43:54.975089+0000","last_peered":"2024-09-13T16:43:54.975089+0000","last_clean":"2024-09-13T16:43:54.975089+0000","last_became_active":"2024-09-13T16:43:54.560480+0000","last_became_peered":"2024-09-13T16:43:54.560480+0000","last_unstale":"2024-09-13T16:43:54.975089+0000","last_undegraded":"2024-09-13T16:43:54.975089+0000","last_fullsized":"2024-09-13T16:43:54.975089+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935689+0000","last_change":"2024-09-13T16:43:54.986459+0000","last_active":"2024-09-13T16:43:56.935689+0000","last_peered":"2024-09-13T16:43:56.935689+0000","last_clean":"2024-09-13T16:43:56.935689+0000","last_became_active":"2024-09-13T16:43:54.986083+0000","last_became_peered":"2024-09-13T16:43:54.986083+0000","last_unstale":"2024-09-13T16:43:56.935689+0000","last_undegraded":"2024-09-13T16:43:56.935689+0000","last_fullsized":"2024-09-13T16:43:56.935689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:51.960459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.978381+0000","last_change":"2024-09-13T16:43:54.565929+0000","last_active":"2024-09-13T16:43:54.978381+0000","last_peered":"2024-09-13T16:43:54.978381+0000","last_clean":"2024-09-13T16:43:54.978381+0000","last_became_active":"2024-09-13T16:43:54.565701+0000","last_became_peered":"2024-09-13T16:43:54.565701+0000","last_unstale":"2024-09-13T16:43:54.978381+0000","last_undegraded":"2024-09-13T16:43:54.978381+0000","last_fullsized":"2024-09-13T16:43:54.978381+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178232+0000","last_change":"2024-09-13T16:43:54.989068+0000","last_active":"2024-09-13T16:43:59.178232+0000","last_peered":"2024-09-13T16:43:59.178232+0000","last_clean":"2024-09-13T16:43:59.178232+0000","last_became_active":"2024-09-13T16:43:54.988779+0000","last_became_peered":"2024-09-13T16:43:54.988779+0000","last_unstale":"2024-09-13T16:43:59.178232+0000","last_undegraded":"2024-09-13T16:43:59.178232+0000","last_fullsized":"2024-09-13T16:43:59.178232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.985188+0000","last_change":"2024-09-13T16:43:54.561487+0000","last_active":"2024-09-13T16:43:54.985188+0000","last_peered":"2024-09-13T16:43:54.985188+0000","last_clean":"2024-09-13T16:43:54.985188+0000","last_became_active":"2024-09-13T16:43:54.561077+0000","last_became_peered":"2024-09-13T16:43:54.561077+0000","last_unstale":"2024-09-13T16:43:54.985188+0000","last_undegraded":"2024-09-13T16:43:54.985188+0000","last_fullsized":"2024-09-13T16:43:54.985188+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872141+0000","last_change":"2024-09-13T16:43:54.999517+0000","last_active":"2024-09-13T16:43:56.872141+0000","last_peered":"2024-09-13T16:43:56.872141+0000","last_clean":"2024-09-13T16:43:56.872141+0000","last_became_active":"2024-09-13T16:43:54.999283+0000","last_became_peered":"2024-09-13T16:43:54.999283+0000","last_unstale":"2024-09-13T16:43:56.872141+0000","last_undegraded":"2024-09-13T16:43:56.872141+0000","last_fullsized":"2024-09-13T16:43:56.872141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:20.207775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.975909+0000","last_change":"2024-09-13T16:43:54.542871+0000","last_active":"2024-09-13T16:43:54.975909+0000","last_peered":"2024-09-13T16:43:54.975909+0000","last_clean":"2024-09-13T16:43:54.975909+0000","last_became_active":"2024-09-13T16:43:54.542548+0000","last_became_peered":"2024-09-13T16:43:54.542548+0000","last_unstale":"2024-09-13T16:43:54.975909+0000","last_undegraded":"2024-09-13T16:43:54.975909+0000","last_fullsized":"2024-09-13T16:43:54.975909+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178181+0000","last_change":"2024-09-13T16:43:54.989168+0000","last_active":"2024-09-13T16:43:59.178181+0000","last_peered":"2024-09-13T16:43:59.178181+0000","last_clean":"2024-09-13T16:43:59.178181+0000","last_became_active":"2024-09-13T16:43:54.988716+0000","last_became_peered":"2024-09-13T16:43:54.988716+0000","last_unstale":"2024-09-13T16:43:59.178181+0000","last_undegraded":"2024-09-13T16:43:59.178181+0000","last_fullsized":"2024-09-13T16:43:59.178181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.978600+0000","last_change":"2024-09-13T16:43:54.551551+0000","last_active":"2024-09-13T16:43:54.978600+0000","last_peered":"2024-09-13T16:43:54.978600+0000","last_clean":"2024-09-13T16:43:54.978600+0000","last_became_active":"2024-09-13T16:43:54.551293+0000","last_became_peered":"2024-09-13T16:43:54.551293+0000","last_unstale":"2024-09-13T16:43:54.978600+0000","last_undegraded":"2024-09-13T16:43:54.978600+0000","last_fullsized":"2024-09-13T16:43:54.978600+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260677+0000","last_change":"2024-09-13T16:43:54.994516+0000","last_active":"2024-09-13T16:43:56.260677+0000","last_peered":"2024-09-13T16:43:56.260677+0000","last_clean":"2024-09-13T16:43:56.260677+0000","last_became_active":"2024-09-13T16:43:54.994245+0000","last_became_peered":"2024-09-13T16:43:54.994245+0000","last_unstale":"2024-09-13T16:43:56.260677+0000","last_undegraded":"2024-09-13T16:43:56.260677+0000","last_fullsized":"2024-09-13T16:43:56.260677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.976834+0000","last_change":"2024-09-13T16:43:54.562232+0000","last_active":"2024-09-13T16:43:54.976834+0000","last_peered":"2024-09-13T16:43:54.976834+0000","last_clean":"2024-09-13T16:43:54.976834+0000","last_became_active":"2024-09-13T16:43:54.561939+0000","last_became_peered":"2024-09-13T16:43:54.561939+0000","last_unstale":"2024-09-13T16:43:54.976834+0000","last_undegraded":"2024-09-13T16:43:54.976834+0000","last_fullsized":"2024-09-13T16:43:54.976834+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178047+0000","last_change":"2024-09-13T16:43:54.988439+0000","last_active":"2024-09-13T16:43:59.178047+0000","last_peered":"2024-09-13T16:43:59.178047+0000","last_clean":"2024-09-13T16:43:59.178047+0000","last_became_active":"2024-09-13T16:43:54.988172+0000","last_became_peered":"2024-09-13T16:43:54.988172+0000","last_unstale":"2024-09-13T16:43:59.178047+0000","last_undegraded":"2024-09-13T16:43:59.178047+0000","last_fullsized":"2024-09-13T16:43:59.178047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.978234+0000","last_change":"2024-09-13T16:43:54.559372+0000","last_active":"2024-09-13T16:43:54.978234+0000","last_peered":"2024-09-13T16:43:54.978234+0000","last_clean":"2024-09-13T16:43:54.978234+0000","last_became_active":"2024-09-13T16:43:54.559111+0000","last_became_peered":"2024-09-13T16:43:54.559111+0000","last_unstale":"2024-09-13T16:43:54.978234+0000","last_undegraded":"2024-09-13T16:43:54.978234+0000","last_fullsized":"2024-09-13T16:43:54.978234+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:40.399883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178451+0000","last_change":"2024-09-13T16:43:54.988313+0000","last_active":"2024-09-13T16:43:59.178451+0000","last_peered":"2024-09-13T16:43:59.178451+0000","last_clean":"2024-09-13T16:43:59.178451+0000","last_became_active":"2024-09-13T16:43:54.988059+0000","last_became_peered":"2024-09-13T16:43:54.988059+0000","last_unstale":"2024-09-13T16:43:59.178451+0000","last_undegraded":"2024-09-13T16:43:59.178451+0000","last_fullsized":"2024-09-13T16:43:59.178451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.977092+0000","last_change":"2024-09-13T16:43:54.566410+0000","last_active":"2024-09-13T16:43:54.977092+0000","last_peered":"2024-09-13T16:43:54.977092+0000","last_clean":"2024-09-13T16:43:54.977092+0000","last_became_active":"2024-09-13T16:43:54.565861+0000","last_became_peered":"2024-09-13T16:43:54.565861+0000","last_unstale":"2024-09-13T16:43:54.977092+0000","last_undegraded":"2024-09-13T16:43:54.977092+0000","last_fullsized":"2024-09-13T16:43:54.977092+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178547+0000","last_change":"2024-09-13T16:43:54.989506+0000","last_active":"2024-09-13T16:43:59.178547+0000","last_peered":"2024-09-13T16:43:59.178547+0000","last_clean":"2024-09-13T16:43:59.178547+0000","last_became_active":"2024-09-13T16:43:54.989233+0000","last_became_peered":"2024-09-13T16:43:54.989233+0000","last_unstale":"2024-09-13T16:43:59.178547+0000","last_undegraded":"2024-09-13T16:43:59.178547+0000","last_fullsized":"2024-09-13T16:43:59.178547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:23.913470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.975177+0000","last_change":"2024-09-13T16:43:54.559723+0000","last_active":"2024-09-13T16:43:54.975177+0000","last_peered":"2024-09-13T16:43:54.975177+0000","last_clean":"2024-09-13T16:43:54.975177+0000","last_became_active":"2024-09-13T16:43:54.559353+0000","last_became_peered":"2024-09-13T16:43:54.559353+0000","last_unstale":"2024-09-13T16:43:54.975177+0000","last_undegraded":"2024-09-13T16:43:54.975177+0000","last_fullsized":"2024-09-13T16:43:54.975177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.995311+0000","last_change":"2024-09-13T16:43:54.992511+0000","last_active":"2024-09-13T16:43:54.995311+0000","last_peered":"2024-09-13T16:43:54.995311+0000","last_clean":"2024-09-13T16:43:54.995311+0000","last_became_active":"2024-09-13T16:43:54.992249+0000","last_became_peered":"2024-09-13T16:43:54.992249+0000","last_unstale":"2024-09-13T16:43:54.995311+0000","last_undegraded":"2024-09-13T16:43:54.995311+0000","last_fullsized":"2024-09-13T16:43:54.995311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.977044+0000","last_change":"2024-09-13T16:43:54.551390+0000","last_active":"2024-09-13T16:43:54.977044+0000","last_peered":"2024-09-13T16:43:54.977044+0000","last_clean":"2024-09-13T16:43:54.977044+0000","last_became_active":"2024-09-13T16:43:54.547074+0000","last_became_peered":"2024-09-13T16:43:54.547074+0000","last_unstale":"2024-09-13T16:43:54.977044+0000","last_undegraded":"2024-09-13T16:43:54.977044+0000","last_fullsized":"2024-09-13T16:43:54.977044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178318+0000","last_change":"2024-09-13T16:43:54.990458+0000","last_active":"2024-09-13T16:43:59.178318+0000","last_peered":"2024-09-13T16:43:59.178318+0000","last_clean":"2024-09-13T16:43:59.178318+0000","last_became_active":"2024-09-13T16:43:54.990196+0000","last_became_peered":"2024-09-13T16:43:54.990196+0000","last_unstale":"2024-09-13T16:43:59.178318+0000","last_undegraded":"2024-09-13T16:43:59.178318+0000","last_fullsized":"2024-09-13T16:43:59.178318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981787+0000","last_change":"2024-09-13T16:43:51.914639+0000","last_active":"2024-09-13T16:43:54.981787+0000","last_peered":"2024-09-13T16:43:54.981787+0000","last_clean":"2024-09-13T16:43:54.981787+0000","last_became_active":"2024-09-13T16:43:51.913797+0000","last_became_peered":"2024-09-13T16:43:51.913797+0000","last_unstale":"2024-09-13T16:43:54.981787+0000","last_undegraded":"2024-09-13T16:43:54.981787+0000","last_fullsized":"2024-09-13T16:43:54.981787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_clean_scrub_stamp":"2024-09-13T16:43:50.498116+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:12.730911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.974632+0000","last_change":"2024-09-13T16:43:54.564153+0000","last_active":"2024-09-13T16:43:54.974632+0000","last_peered":"2024-09-13T16:43:54.974632+0000","last_clean":"2024-09-13T16:43:54.974632+0000","last_became_active":"2024-09-13T16:43:54.563949+0000","last_became_peered":"2024-09-13T16:43:54.563949+0000","last_unstale":"2024-09-13T16:43:54.974632+0000","last_undegraded":"2024-09-13T16:43:54.974632+0000","last_fullsized":"2024-09-13T16:43:54.974632+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508836+0000","last_change":"2024-09-13T16:43:54.997576+0000","last_active":"2024-09-13T16:43:56.508836+0000","last_peered":"2024-09-13T16:43:56.508836+0000","last_clean":"2024-09-13T16:43:56.508836+0000","last_became_active":"2024-09-13T16:43:54.997315+0000","last_became_peered":"2024-09-13T16:43:54.997315+0000","last_unstale":"2024-09-13T16:43:56.508836+0000","last_undegraded":"2024-09-13T16:43:56.508836+0000","last_fullsized":"2024-09-13T16:43:56.508836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:17.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981701+0000","last_change":"2024-09-13T16:43:54.564804+0000","last_active":"2024-09-13T16:43:54.981701+0000","last_peered":"2024-09-13T16:43:54.981701+0000","last_clean":"2024-09-13T16:43:54.981701+0000","last_became_active":"2024-09-13T16:43:54.564605+0000","last_became_peered":"2024-09-13T16:43:54.564605+0000","last_unstale":"2024-09-13T16:43:54.981701+0000","last_undegraded":"2024-09-13T16:43:54.981701+0000","last_fullsized":"2024-09-13T16:43:54.981701+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.871801+0000","last_change":"2024-09-13T16:43:54.997884+0000","last_active":"2024-09-13T16:43:56.871801+0000","last_peered":"2024-09-13T16:43:56.871801+0000","last_clean":"2024-09-13T16:43:56.871801+0000","last_became_active":"2024-09-13T16:43:54.997510+0000","last_became_peered":"2024-09-13T16:43:54.997510+0000","last_unstale":"2024-09-13T16:43:56.871801+0000","last_undegraded":"2024-09-13T16:43:56.871801+0000","last_fullsized":"2024-09-13T16:43:56.871801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.976301+0000","last_change":"2024-09-13T16:43:54.566294+0000","last_active":"2024-09-13T16:43:54.976301+0000","last_peered":"2024-09-13T16:43:54.976301+0000","last_clean":"2024-09-13T16:43:54.976301+0000","last_became_active":"2024-09-13T16:43:54.566023+0000","last_became_peered":"2024-09-13T16:43:54.566023+0000","last_unstale":"2024-09-13T16:43:54.976301+0000","last_undegraded":"2024-09-13T16:43:54.976301+0000","last_fullsized":"2024-09-13T16:43:54.976301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975316+0000","last_change":"2024-09-13T16:43:54.981116+0000","last_active":"2024-09-13T16:43:55.975316+0000","last_peered":"2024-09-13T16:43:55.975316+0000","last_clean":"2024-09-13T16:43:55.975316+0000","last_became_active":"2024-09-13T16:43:54.980766+0000","last_became_peered":"2024-09-13T16:43:54.980766+0000","last_unstale":"2024-09-13T16:43:55.975316+0000","last_undegraded":"2024-09-13T16:43:55.975316+0000","last_fullsized":"2024-09-13T16:43:55.975316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.976007+0000","last_change":"2024-09-13T16:43:54.563716+0000","last_active":"2024-09-13T16:43:54.976007+0000","last_peered":"2024-09-13T16:43:54.976007+0000","last_clean":"2024-09-13T16:43:54.976007+0000","last_became_active":"2024-09-13T16:43:54.563522+0000","last_became_peered":"2024-09-13T16:43:54.563522+0000","last_unstale":"2024-09-13T16:43:54.976007+0000","last_undegraded":"2024-09-13T16:43:54.976007+0000","last_fullsized":"2024-09-13T16:43:54.976007+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935617+0000","last_change":"2024-09-13T16:43:54.999051+0000","last_active":"2024-09-13T16:43:56.935617+0000","last_peered":"2024-09-13T16:43:56.935617+0000","last_clean":"2024-09-13T16:43:56.935617+0000","last_became_active":"2024-09-13T16:43:54.998803+0000","last_became_peered":"2024-09-13T16:43:54.998803+0000","last_unstale":"2024-09-13T16:43:56.935617+0000","last_undegraded":"2024-09-13T16:43:56.935617+0000","last_fullsized":"2024-09-13T16:43:56.935617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.974615+0000","last_change":"2024-09-13T16:43:54.569945+0000","last_active":"2024-09-13T16:43:54.974615+0000","last_peered":"2024-09-13T16:43:54.974615+0000","last_clean":"2024-09-13T16:43:54.974615+0000","last_became_active":"2024-09-13T16:43:54.568846+0000","last_became_peered":"2024-09-13T16:43:54.568846+0000","last_unstale":"2024-09-13T16:43:54.974615+0000","last_undegraded":"2024-09-13T16:43:54.974615+0000","last_fullsized":"2024-09-13T16:43:54.974615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.995256+0000","last_change":"2024-09-13T16:43:54.992082+0000","last_active":"2024-09-13T16:43:54.995256+0000","last_peered":"2024-09-13T16:43:54.995256+0000","last_clean":"2024-09-13T16:43:54.995256+0000","last_became_active":"2024-09-13T16:43:54.991659+0000","last_became_peered":"2024-09-13T16:43:54.991659+0000","last_unstale":"2024-09-13T16:43:54.995256+0000","last_undegraded":"2024-09-13T16:43:54.995256+0000","last_fullsized":"2024-09-13T16:43:54.995256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981345+0000","last_change":"2024-09-13T16:43:54.569111+0000","last_active":"2024-09-13T16:43:54.981345+0000","last_peered":"2024-09-13T16:43:54.981345+0000","last_clean":"2024-09-13T16:43:54.981345+0000","last_became_active":"2024-09-13T16:43:54.568903+0000","last_became_peered":"2024-09-13T16:43:54.568903+0000","last_unstale":"2024-09-13T16:43:54.981345+0000","last_undegraded":"2024-09-13T16:43:54.981345+0000","last_fullsized":"2024-09-13T16:43:54.981345+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975475+0000","last_change":"2024-09-13T16:43:54.980100+0000","last_active":"2024-09-13T16:43:55.975475+0000","last_peered":"2024-09-13T16:43:55.975475+0000","last_clean":"2024-09-13T16:43:55.975475+0000","last_became_active":"2024-09-13T16:43:54.979584+0000","last_became_peered":"2024-09-13T16:43:54.979584+0000","last_unstale":"2024-09-13T16:43:55.975475+0000","last_undegraded":"2024-09-13T16:43:55.975475+0000","last_fullsized":"2024-09-13T16:43:55.975475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.975232+0000","last_change":"2024-09-13T16:43:54.560710+0000","last_active":"2024-09-13T16:43:54.975232+0000","last_peered":"2024-09-13T16:43:54.975232+0000","last_clean":"2024-09-13T16:43:54.975232+0000","last_became_active":"2024-09-13T16:43:54.560030+0000","last_became_peered":"2024-09-13T16:43:54.560030+0000","last_unstale":"2024-09-13T16:43:54.975232+0000","last_undegraded":"2024-09-13T16:43:54.975232+0000","last_fullsized":"2024-09-13T16:43:54.975232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.509048+0000","last_change":"2024-09-13T16:43:54.990812+0000","last_active":"2024-09-13T16:43:56.509048+0000","last_peered":"2024-09-13T16:43:56.509048+0000","last_clean":"2024-09-13T16:43:56.509048+0000","last_became_active":"2024-09-13T16:43:54.990440+0000","last_became_peered":"2024-09-13T16:43:54.990440+0000","last_unstale":"2024-09-13T16:43:56.509048+0000","last_undegraded":"2024-09-13T16:43:56.509048+0000","last_fullsized":"2024-09-13T16:43:56.509048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.978914+0000","last_change":"2024-09-13T16:43:54.558439+0000","last_active":"2024-09-13T16:43:54.978914+0000","last_peered":"2024-09-13T16:43:54.978914+0000","last_clean":"2024-09-13T16:43:54.978914+0000","last_became_active":"2024-09-13T16:43:54.558194+0000","last_became_peered":"2024-09-13T16:43:54.558194+0000","last_unstale":"2024-09-13T16:43:54.978914+0000","last_undegraded":"2024-09-13T16:43:54.978914+0000","last_fullsized":"2024-09-13T16:43:54.978914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872087+0000","last_change":"2024-09-13T16:43:54.995534+0000","last_active":"2024-09-13T16:43:56.872087+0000","last_peered":"2024-09-13T16:43:56.872087+0000","last_clean":"2024-09-13T16:43:56.872087+0000","last_became_active":"2024-09-13T16:43:54.995397+0000","last_became_peered":"2024-09-13T16:43:54.995397+0000","last_unstale":"2024-09-13T16:43:56.872087+0000","last_undegraded":"2024-09-13T16:43:56.872087+0000","last_fullsized":"2024-09-13T16:43:56.872087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:19.697247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981614+0000","last_change":"2024-09-13T16:43:54.569085+0000","last_active":"2024-09-13T16:43:54.981614+0000","last_peered":"2024-09-13T16:43:54.981614+0000","last_clean":"2024-09-13T16:43:54.981614+0000","last_became_active":"2024-09-13T16:43:54.568892+0000","last_became_peered":"2024-09-13T16:43:54.568892+0000","last_unstale":"2024-09-13T16:43:54.981614+0000","last_undegraded":"2024-09-13T16:43:54.981614+0000","last_fullsized":"2024-09-13T16:43:54.981614+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935459+0000","last_change":"2024-09-13T16:43:54.984120+0000","last_active":"2024-09-13T16:43:56.935459+0000","last_peered":"2024-09-13T16:43:56.935459+0000","last_clean":"2024-09-13T16:43:56.935459+0000","last_became_active":"2024-09-13T16:43:54.983886+0000","last_became_peered":"2024-09-13T16:43:54.983886+0000","last_unstale":"2024-09-13T16:43:56.935459+0000","last_undegraded":"2024-09-13T16:43:56.935459+0000","last_fullsized":"2024-09-13T16:43:56.935459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.973946+0000","last_change":"2024-09-13T16:43:54.555092+0000","last_active":"2024-09-13T16:43:54.973946+0000","last_peered":"2024-09-13T16:43:54.973946+0000","last_clean":"2024-09-13T16:43:54.973946+0000","last_became_active":"2024-09-13T16:43:54.554230+0000","last_became_peered":"2024-09-13T16:43:54.554230+0000","last_unstale":"2024-09-13T16:43:54.973946+0000","last_undegraded":"2024-09-13T16:43:54.973946+0000","last_fullsized":"2024-09-13T16:43:54.973946+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508681+0000","last_change":"2024-09-13T16:43:54.984588+0000","last_active":"2024-09-13T16:43:56.508681+0000","last_peered":"2024-09-13T16:43:56.508681+0000","last_clean":"2024-09-13T16:43:56.508681+0000","last_became_active":"2024-09-13T16:43:54.984221+0000","last_became_peered":"2024-09-13T16:43:54.984221+0000","last_unstale":"2024-09-13T16:43:56.508681+0000","last_undegraded":"2024-09-13T16:43:56.508681+0000","last_fullsized":"2024-09-13T16:43:56.508681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.973687+0000","last_change":"2024-09-13T16:43:54.562646+0000","last_active":"2024-09-13T16:43:54.973687+0000","last_peered":"2024-09-13T16:43:54.973687+0000","last_clean":"2024-09-13T16:43:54.973687+0000","last_became_active":"2024-09-13T16:43:54.556888+0000","last_became_peered":"2024-09-13T16:43:54.556888+0000","last_unstale":"2024-09-13T16:43:54.973687+0000","last_undegraded":"2024-09-13T16:43:54.973687+0000","last_fullsized":"2024-09-13T16:43:54.973687+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508792+0000","last_change":"2024-09-13T16:43:54.999328+0000","last_active":"2024-09-13T16:43:56.508792+0000","last_peered":"2024-09-13T16:43:56.508792+0000","last_clean":"2024-09-13T16:43:56.508792+0000","last_became_active":"2024-09-13T16:43:54.999101+0000","last_became_peered":"2024-09-13T16:43:54.999101+0000","last_unstale":"2024-09-13T16:43:56.508792+0000","last_undegraded":"2024-09-13T16:43:56.508792+0000","last_fullsized":"2024-09-13T16:43:56.508792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.979629+0000","last_change":"2024-09-13T16:43:54.559214+0000","last_active":"2024-09-13T16:43:54.979629+0000","last_peered":"2024-09-13T16:43:54.979629+0000","last_clean":"2024-09-13T16:43:54.979629+0000","last_became_active":"2024-09-13T16:43:54.558233+0000","last_became_peered":"2024-09-13T16:43:54.558233+0000","last_unstale":"2024-09-13T16:43:54.979629+0000","last_undegraded":"2024-09-13T16:43:54.979629+0000","last_fullsized":"2024-09-13T16:43:54.979629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260789+0000","last_change":"2024-09-13T16:43:54.986565+0000","last_active":"2024-09-13T16:43:56.260789+0000","last_peered":"2024-09-13T16:43:56.260789+0000","last_clean":"2024-09-13T16:43:56.260789+0000","last_became_active":"2024-09-13T16:43:54.986084+0000","last_became_peered":"2024-09-13T16:43:54.986084+0000","last_unstale":"2024-09-13T16:43:56.260789+0000","last_undegraded":"2024-09-13T16:43:56.260789+0000","last_fullsized":"2024-09-13T16:43:56.260789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.989817+0000","last_change":"2024-09-13T16:43:54.982611+0000","last_active":"2024-09-13T16:43:54.989817+0000","last_peered":"2024-09-13T16:43:54.989817+0000","last_clean":"2024-09-13T16:43:54.989817+0000","last_became_active":"2024-09-13T16:43:54.982338+0000","last_became_peered":"2024-09-13T16:43:54.982338+0000","last_unstale":"2024-09-13T16:43:54.989817+0000","last_undegraded":"2024-09-13T16:43:54.989817+0000","last_fullsized":"2024-09-13T16:43:54.989817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.979247+0000","last_change":"2024-09-13T16:43:54.558799+0000","last_active":"2024-09-13T16:43:54.979247+0000","last_peered":"2024-09-13T16:43:54.979247+0000","last_clean":"2024-09-13T16:43:54.979247+0000","last_became_active":"2024-09-13T16:43:54.558332+0000","last_became_peered":"2024-09-13T16:43:54.558332+0000","last_unstale":"2024-09-13T16:43:54.979247+0000","last_undegraded":"2024-09-13T16:43:54.979247+0000","last_fullsized":"2024-09-13T16:43:54.979247+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:19.201506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.993116+0000","last_change":"2024-09-13T16:43:54.984150+0000","last_active":"2024-09-13T16:43:54.993116+0000","last_peered":"2024-09-13T16:43:54.993116+0000","last_clean":"2024-09-13T16:43:54.993116+0000","last_became_active":"2024-09-13T16:43:54.983881+0000","last_became_peered":"2024-09-13T16:43:54.983881+0000","last_unstale":"2024-09-13T16:43:54.993116+0000","last_undegraded":"2024-09-13T16:43:54.993116+0000","last_fullsized":"2024-09-13T16:43:54.993116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:29.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.980937+0000","last_change":"2024-09-13T16:43:54.559551+0000","last_active":"2024-09-13T16:43:54.980937+0000","last_peered":"2024-09-13T16:43:54.980937+0000","last_clean":"2024-09-13T16:43:54.980937+0000","last_became_active":"2024-09-13T16:43:54.559290+0000","last_became_peered":"2024-09-13T16:43:54.559290+0000","last_unstale":"2024-09-13T16:43:54.980937+0000","last_undegraded":"2024-09-13T16:43:54.980937+0000","last_fullsized":"2024-09-13T16:43:54.980937+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872084+0000","last_change":"2024-09-13T16:43:55.000783+0000","last_active":"2024-09-13T16:43:56.872084+0000","last_peered":"2024-09-13T16:43:56.872084+0000","last_clean":"2024-09-13T16:43:56.872084+0000","last_became_active":"2024-09-13T16:43:55.000196+0000","last_became_peered":"2024-09-13T16:43:55.000196+0000","last_unstale":"2024-09-13T16:43:56.872084+0000","last_undegraded":"2024-09-13T16:43:56.872084+0000","last_fullsized":"2024-09-13T16:43:56.872084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.986648+0000","last_change":"2024-09-13T16:43:54.559886+0000","last_active":"2024-09-13T16:43:54.986648+0000","last_peered":"2024-09-13T16:43:54.986648+0000","last_clean":"2024-09-13T16:43:54.986648+0000","last_became_active":"2024-09-13T16:43:54.559586+0000","last_became_peered":"2024-09-13T16:43:54.559586+0000","last_unstale":"2024-09-13T16:43:54.986648+0000","last_undegraded":"2024-09-13T16:43:54.986648+0000","last_fullsized":"2024-09-13T16:43:54.986648+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508978+0000","last_change":"2024-09-13T16:43:54.996099+0000","last_active":"2024-09-13T16:43:56.508978+0000","last_peered":"2024-09-13T16:43:56.508978+0000","last_clean":"2024-09-13T16:43:56.508978+0000","last_became_active":"2024-09-13T16:43:54.995802+0000","last_became_peered":"2024-09-13T16:43:54.995802+0000","last_unstale":"2024-09-13T16:43:56.508978+0000","last_undegraded":"2024-09-13T16:43:56.508978+0000","last_fullsized":"2024-09-13T16:43:56.508978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.983458+0000","last_change":"2024-09-13T16:43:54.562699+0000","last_active":"2024-09-13T16:43:54.983458+0000","last_peered":"2024-09-13T16:43:54.983458+0000","last_clean":"2024-09-13T16:43:54.983458+0000","last_became_active":"2024-09-13T16:43:54.560552+0000","last_became_peered":"2024-09-13T16:43:54.560552+0000","last_unstale":"2024-09-13T16:43:54.983458+0000","last_undegraded":"2024-09-13T16:43:54.983458+0000","last_fullsized":"2024-09-13T16:43:54.983458+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:11.032382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":654966,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574850,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":176128,"data_stored":57209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":70284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":266240,"data_stored":660247,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:44:01.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:44:01.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.354+0000 7fb170400640 1 Processor -- start 2024-09-13T16:44:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.354+0000 7fb170400640 1 -- start start 2024-09-13T16:44:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.355+0000 7fb170400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1680c5bd0 0x7fb168156150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.355+0000 7fb170400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1680c3670 0x7fb1680c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.355+0000 7fb170400640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb16814ca00 con 0x7fb1680c1ea0 2024-09-13T16:44:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.355+0000 7fb170400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1680c2dc0 con 0x7fb1680c3670 2024-09-13T16:44:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.355+0000 7fb170400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb168155780 con 0x7fb1680c5bd0 2024-09-13T16:44:01.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.355+0000 7fb16d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1680c3670 0x7fb1680c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:01.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.355+0000 7fb16d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1680c3670 0x7fb1680c1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41354/0 (socket says 172.21.15.28:41354) 2024-09-13T16:44:01.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.355+0000 7fb16d000640 1 -- 172.21.15.28:0/2736632441 learned_addr learned my addr 172.21.15.28:0/2736632441 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:01.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.355+0000 7fb16da00640 1 --2- 172.21.15.28:0/2736632441 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1680c5bd0 0x7fb168156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:01.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.356+0000 7fb16d000640 1 -- 172.21.15.28:0/2736632441 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1680c5bd0 msgr2=0x7fb168156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:01.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.356+0000 7fb16d000640 1 --2- 172.21.15.28:0/2736632441 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1680c5bd0 0x7fb168156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.356+0000 7fb16d000640 1 -- 172.21.15.28:0/2736632441 >> v1:172.21.15.191:6789/0 conn(0x7fb1680c1ea0 legacy=0x7fb1680c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:01.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.356+0000 7fb16d000640 1 -- 172.21.15.28:0/2736632441 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb168155990 con 0x7fb1680c3670 2024-09-13T16:44:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.357+0000 7fb16d000640 1 --2- 172.21.15.28:0/2736632441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1680c3670 0x7fb1680c1960 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb154009870 tx=0x7fb15402c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f6c14d423e88de0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.357+0000 7fb167e00640 1 -- 172.21.15.28:0/2736632441 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb154031710 con 0x7fb1680c3670 2024-09-13T16:44:01.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.357+0000 7fb167e00640 1 -- 172.21.15.28:0/2736632441 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15402f050 con 0x7fb1680c3670 2024-09-13T16:44:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.357+0000 7fb167e00640 1 -- 172.21.15.28:0/2736632441 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb154031a90 con 0x7fb1680c3670 2024-09-13T16:44:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.357+0000 7fb170400640 1 -- 172.21.15.28:0/2736632441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1680c3670 msgr2=0x7fb1680c1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.357+0000 7fb170400640 1 --2- 172.21.15.28:0/2736632441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1680c3670 0x7fb1680c1960 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb154009870 tx=0x7fb15402c510 comp rx=0 tx=0).stop 2024-09-13T16:44:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.358+0000 7fb170400640 1 -- 172.21.15.28:0/2736632441 shutdown_connections 2024-09-13T16:44:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.358+0000 7fb170400640 1 --2- 172.21.15.28:0/2736632441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1680c3670 0x7fb1680c1960 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.358+0000 7fb170400640 1 --2- 172.21.15.28:0/2736632441 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1680c5bd0 0x7fb168156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.358+0000 7fb170400640 1 -- 172.21.15.28:0/2736632441 >> 172.21.15.28:0/2736632441 conn(0x7fb168066040 msgr2=0x7fb168069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.358+0000 7fb170400640 1 -- 172.21.15.28:0/2736632441 shutdown_connections 2024-09-13T16:44:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.358+0000 7fb170400640 1 -- 172.21.15.28:0/2736632441 wait complete. 2024-09-13T16:44:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.359+0000 7fb170400640 1 Processor -- start 2024-09-13T16:44:01.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.359+0000 7fb170400640 1 -- start start 2024-09-13T16:44:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.360+0000 7fb170400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1680c1ea0 0x7fb1680bd800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.360+0000 7fb170400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1680c3670 0x7fb168153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.360+0000 7fb170400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1680c5bd0 0x7fb16814bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.360+0000 7fb170400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1680bfca0 con 0x7fb1680c1ea0 2024-09-13T16:44:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.360+0000 7fb170400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1680be360 con 0x7fb1680c5bd0 2024-09-13T16:44:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.360+0000 7fb170400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1680bdd40 con 0x7fb1680c3670 2024-09-13T16:44:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.361+0000 7fb16da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1680c1ea0 0x7fb1680bd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.361+0000 7fb16da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1680c1ea0 0x7fb1680bd800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41364/0 (socket says 172.21.15.28:41364) 2024-09-13T16:44:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.361+0000 7fb16da00640 1 -- 172.21.15.28:0/2194616632 learned_addr learned my addr 172.21.15.28:0/2194616632 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.361+0000 7fb16d000640 1 --2- 172.21.15.28:0/2194616632 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1680c3670 0x7fb168153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.361+0000 7fb16e400640 1 --2- 172.21.15.28:0/2194616632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1680c5bd0 0x7fb16814bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.361+0000 7fb16da00640 1 -- 172.21.15.28:0/2194616632 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1680c3670 msgr2=0x7fb168153b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.361+0000 7fb16da00640 1 --2- 172.21.15.28:0/2194616632 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1680c3670 0x7fb168153b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.361+0000 7fb16da00640 1 -- 172.21.15.28:0/2194616632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1680c5bd0 msgr2=0x7fb16814bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.361+0000 7fb16da00640 1 --2- 172.21.15.28:0/2194616632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1680c5bd0 0x7fb16814bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.361+0000 7fb16da00640 1 -- 172.21.15.28:0/2194616632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1540094d0 con 0x7fb1680c1ea0 2024-09-13T16:44:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.361+0000 7fb16e400640 1 --2- 172.21.15.28:0/2194616632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1680c5bd0 0x7fb16814bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.361+0000 7fb16da00640 1 --2- 172.21.15.28:0/2194616632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1680c1ea0 0x7fb1680bd800 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb15c00e870 tx=0x7fb15c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.361+0000 7fb165600640 1 -- 172.21.15.28:0/2194616632 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c0094f0 con 0x7fb1680c1ea0 2024-09-13T16:44:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.361+0000 7fb165600640 1 -- 172.21.15.28:0/2194616632 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c00c940 con 0x7fb1680c1ea0 2024-09-13T16:44:01.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.361+0000 7fb165600640 1 -- 172.21.15.28:0/2194616632 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c00ccf0 con 0x7fb1680c1ea0 2024-09-13T16:44:01.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.361+0000 7fb170400640 1 -- 172.21.15.28:0/2194616632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1680bdfb0 con 0x7fb1680c1ea0 2024-09-13T16:44:01.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.362+0000 7fb170400640 1 -- 172.21.15.28:0/2194616632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1680be850 con 0x7fb1680c1ea0 2024-09-13T16:44:01.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.363+0000 7fb170400640 1 -- 172.21.15.28:0/2194616632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb168070680 con 0x7fb1680c1ea0 2024-09-13T16:44:01.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.363+0000 7fb165600640 1 -- 172.21.15.28:0/2194616632 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb15c013470 con 0x7fb1680c1ea0 2024-09-13T16:44:01.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.364+0000 7fb165600640 1 --2- 172.21.15.28:0/2194616632 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb14806ac00 0x7fb14806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:01.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.364+0000 7fb165600640 1 -- 172.21.15.28:0/2194616632 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb15c089510 con 0x7fb1680c1ea0 2024-09-13T16:44:01.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.365+0000 7fb16d000640 1 --2- 172.21.15.28:0/2194616632 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb14806ac00 0x7fb14806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:01.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.365+0000 7fb16d000640 1 --2- 172.21.15.28:0/2194616632 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb14806ac00 0x7fb14806d0c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb154035000 tx=0x7fb154008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:01.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.368+0000 7fb165600640 1 -- 172.21.15.28:0/2194616632 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb15c054670 con 0x7fb1680c1ea0 2024-09-13T16:44:01.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.542+0000 7fb170400640 1 -- 172.21.15.28:0/2194616632 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb1680c3e40 con 0x7fb14806ac00 2024-09-13T16:44:01.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.549+0000 7fb165600640 1 -- 172.21.15.28:0/2194616632 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264611 (secure 0 0 0) 0x7fb154031800 con 0x7fb14806ac00 2024-09-13T16:44:01.553 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:01.555 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T16:44:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.553+0000 7fb170400640 1 -- 172.21.15.28:0/2194616632 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb14806ac00 msgr2=0x7fb14806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.553+0000 7fb170400640 1 --2- 172.21.15.28:0/2194616632 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb14806ac00 0x7fb14806d0c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb154035000 tx=0x7fb154008030 comp rx=0 tx=0).stop 2024-09-13T16:44:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.553+0000 7fb170400640 1 -- 172.21.15.28:0/2194616632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1680c1ea0 msgr2=0x7fb1680bd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:01.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.553+0000 7fb170400640 1 --2- 172.21.15.28:0/2194616632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1680c1ea0 0x7fb1680bd800 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb15c00e870 tx=0x7fb15c00ed30 comp rx=0 tx=0).stop 2024-09-13T16:44:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.553+0000 7fb170400640 1 -- 172.21.15.28:0/2194616632 shutdown_connections 2024-09-13T16:44:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.553+0000 7fb170400640 1 --2- 172.21.15.28:0/2194616632 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb14806ac00 0x7fb14806d0c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.553+0000 7fb170400640 1 --2- 172.21.15.28:0/2194616632 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb1680c5bd0 0x7fb16814bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.553+0000 7fb170400640 1 --2- 172.21.15.28:0/2194616632 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb1680c3670 0x7fb168153b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.553+0000 7fb170400640 1 --2- 172.21.15.28:0/2194616632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1680c1ea0 0x7fb1680bd800 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.553+0000 7fb170400640 1 -- 172.21.15.28:0/2194616632 >> 172.21.15.28:0/2194616632 conn(0x7fb168066040 msgr2=0x7fb1680c6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.553+0000 7fb170400640 1 -- 172.21.15.28:0/2194616632 shutdown_connections 2024-09-13T16:44:01.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.554+0000 7fb170400640 1 -- 172.21.15.28:0/2194616632 wait complete. 2024-09-13T16:44:01.570 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-13T16:43:59.591349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221988,"kb_used_data":1680,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754752732,"statfs":{"total":773094113280,"available":772866797568,"internally_reserved":0,"allocated":1720320,"data_stored":1726298,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12743,"internal_metadata":225431097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.058679"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178474+0000","last_change":"2024-09-13T16:43:54.987241+0000","last_active":"2024-09-13T16:43:59.178474+0000","last_peered":"2024-09-13T16:43:59.178474+0000","last_clean":"2024-09-13T16:43:59.178474+0000","last_became_active":"2024-09-13T16:43:54.986988+0000","last_became_peered":"2024-09-13T16:43:54.986988+0000","last_unstale":"2024-09-13T16:43:59.178474+0000","last_undegraded":"2024-09-13T16:43:59.178474+0000","last_fullsized":"2024-09-13T16:43:59.178474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975685+0000","last_change":"2024-09-13T16:43:55.001141+0000","last_active":"2024-09-13T16:43:55.975685+0000","last_peered":"2024-09-13T16:43:55.975685+0000","last_clean":"2024-09-13T16:43:55.975685+0000","last_became_active":"2024-09-13T16:43:55.000757+0000","last_became_peered":"2024-09-13T16:43:55.000757+0000","last_unstale":"2024-09-13T16:43:55.975685+0000","last_undegraded":"2024-09-13T16:43:55.975685+0000","last_fullsized":"2024-09-13T16:43:55.975685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872108+0000","last_change":"2024-09-13T16:43:54.986925+0000","last_active":"2024-09-13T16:43:56.872108+0000","last_peered":"2024-09-13T16:43:56.872108+0000","last_clean":"2024-09-13T16:43:56.872108+0000","last_became_active":"2024-09-13T16:43:54.986198+0000","last_became_peered":"2024-09-13T16:43:54.986198+0000","last_unstale":"2024-09-13T16:43:56.872108+0000","last_undegraded":"2024-09-13T16:43:56.872108+0000","last_fullsized":"2024-09-13T16:43:56.872108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:03.127550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.995272+0000","last_change":"2024-09-13T16:43:54.992249+0000","last_active":"2024-09-13T16:43:54.995272+0000","last_peered":"2024-09-13T16:43:54.995272+0000","last_clean":"2024-09-13T16:43:54.995272+0000","last_became_active":"2024-09-13T16:43:54.991804+0000","last_became_peered":"2024-09-13T16:43:54.991804+0000","last_unstale":"2024-09-13T16:43:54.995272+0000","last_undegraded":"2024-09-13T16:43:54.995272+0000","last_fullsized":"2024-09-13T16:43:54.995272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178163+0000","last_change":"2024-09-13T16:43:54.987564+0000","last_active":"2024-09-13T16:43:59.178163+0000","last_peered":"2024-09-13T16:43:59.178163+0000","last_clean":"2024-09-13T16:43:59.178163+0000","last_became_active":"2024-09-13T16:43:54.987328+0000","last_became_peered":"2024-09-13T16:43:54.987328+0000","last_unstale":"2024-09-13T16:43:59.178163+0000","last_undegraded":"2024-09-13T16:43:59.178163+0000","last_fullsized":"2024-09-13T16:43:59.178163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:36.006730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.005629+0000","last_change":"2024-09-13T16:43:55.000019+0000","last_active":"2024-09-13T16:43:55.005629+0000","last_peered":"2024-09-13T16:43:55.005629+0000","last_clean":"2024-09-13T16:43:55.005629+0000","last_became_active":"2024-09-13T16:43:54.999752+0000","last_became_peered":"2024-09-13T16:43:54.999752+0000","last_unstale":"2024-09-13T16:43:55.005629+0000","last_undegraded":"2024-09-13T16:43:55.005629+0000","last_fullsized":"2024-09-13T16:43:55.005629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.177985+0000","last_change":"2024-09-13T16:43:54.999533+0000","last_active":"2024-09-13T16:43:59.177985+0000","last_peered":"2024-09-13T16:43:59.177985+0000","last_clean":"2024-09-13T16:43:59.177985+0000","last_became_active":"2024-09-13T16:43:54.999350+0000","last_became_peered":"2024-09-13T16:43:54.999350+0000","last_unstale":"2024-09-13T16:43:59.177985+0000","last_undegraded":"2024-09-13T16:43:59.177985+0000","last_fullsized":"2024-09-13T16:43:59.177985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:29.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508531+0000","last_change":"2024-09-13T16:43:54.998526+0000","last_active":"2024-09-13T16:43:56.508531+0000","last_peered":"2024-09-13T16:43:56.508531+0000","last_clean":"2024-09-13T16:43:56.508531+0000","last_became_active":"2024-09-13T16:43:54.996309+0000","last_became_peered":"2024-09-13T16:43:54.996309+0000","last_unstale":"2024-09-13T16:43:56.508531+0000","last_undegraded":"2024-09-13T16:43:56.508531+0000","last_fullsized":"2024-09-13T16:43:56.508531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178473+0000","last_change":"2024-09-13T16:43:54.993786+0000","last_active":"2024-09-13T16:43:59.178473+0000","last_peered":"2024-09-13T16:43:59.178473+0000","last_clean":"2024-09-13T16:43:59.178473+0000","last_became_active":"2024-09-13T16:43:54.993475+0000","last_became_peered":"2024-09-13T16:43:54.993475+0000","last_unstale":"2024-09-13T16:43:59.178473+0000","last_undegraded":"2024-09-13T16:43:59.178473+0000","last_fullsized":"2024-09-13T16:43:59.178473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260988+0000","last_change":"2024-09-13T16:43:54.986272+0000","last_active":"2024-09-13T16:43:56.260988+0000","last_peered":"2024-09-13T16:43:56.260988+0000","last_clean":"2024-09-13T16:43:56.260988+0000","last_became_active":"2024-09-13T16:43:54.985979+0000","last_became_peered":"2024-09-13T16:43:54.985979+0000","last_unstale":"2024-09-13T16:43:56.260988+0000","last_undegraded":"2024-09-13T16:43:56.260988+0000","last_fullsized":"2024-09-13T16:43:56.260988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260766+0000","last_change":"2024-09-13T16:43:54.986207+0000","last_active":"2024-09-13T16:43:56.260766+0000","last_peered":"2024-09-13T16:43:56.260766+0000","last_clean":"2024-09-13T16:43:56.260766+0000","last_became_active":"2024-09-13T16:43:54.985948+0000","last_became_peered":"2024-09-13T16:43:54.985948+0000","last_unstale":"2024-09-13T16:43:56.260766+0000","last_undegraded":"2024-09-13T16:43:56.260766+0000","last_fullsized":"2024-09-13T16:43:56.260766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935591+0000","last_change":"2024-09-13T16:43:54.986636+0000","last_active":"2024-09-13T16:43:56.935591+0000","last_peered":"2024-09-13T16:43:56.935591+0000","last_clean":"2024-09-13T16:43:56.935591+0000","last_became_active":"2024-09-13T16:43:54.986161+0000","last_became_peered":"2024-09-13T16:43:54.986161+0000","last_unstale":"2024-09-13T16:43:56.935591+0000","last_undegraded":"2024-09-13T16:43:56.935591+0000","last_fullsized":"2024-09-13T16:43:56.935591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:09.340330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508720+0000","last_change":"2024-09-13T16:43:54.997783+0000","last_active":"2024-09-13T16:43:56.508720+0000","last_peered":"2024-09-13T16:43:56.508720+0000","last_clean":"2024-09-13T16:43:56.508720+0000","last_became_active":"2024-09-13T16:43:54.997162+0000","last_became_peered":"2024-09-13T16:43:54.997162+0000","last_unstale":"2024-09-13T16:43:56.508720+0000","last_undegraded":"2024-09-13T16:43:56.508720+0000","last_fullsized":"2024-09-13T16:43:56.508720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872216+0000","last_change":"2024-09-13T16:43:54.995262+0000","last_active":"2024-09-13T16:43:56.872216+0000","last_peered":"2024-09-13T16:43:56.872216+0000","last_clean":"2024-09-13T16:43:56.872216+0000","last_became_active":"2024-09-13T16:43:54.994891+0000","last_became_peered":"2024-09-13T16:43:54.994891+0000","last_unstale":"2024-09-13T16:43:56.872216+0000","last_undegraded":"2024-09-13T16:43:56.872216+0000","last_fullsized":"2024-09-13T16:43:56.872216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975286+0000","last_change":"2024-09-13T16:43:54.981534+0000","last_active":"2024-09-13T16:43:55.975286+0000","last_peered":"2024-09-13T16:43:55.975286+0000","last_clean":"2024-09-13T16:43:55.975286+0000","last_became_active":"2024-09-13T16:43:54.981191+0000","last_became_peered":"2024-09-13T16:43:54.981191+0000","last_unstale":"2024-09-13T16:43:55.975286+0000","last_undegraded":"2024-09-13T16:43:55.975286+0000","last_fullsized":"2024-09-13T16:43:55.975286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.997667+0000","last_change":"2024-09-13T16:43:54.994157+0000","last_active":"2024-09-13T16:43:54.997667+0000","last_peered":"2024-09-13T16:43:54.997667+0000","last_clean":"2024-09-13T16:43:54.997667+0000","last_became_active":"2024-09-13T16:43:54.993915+0000","last_became_peered":"2024-09-13T16:43:54.993915+0000","last_unstale":"2024-09-13T16:43:54.997667+0000","last_undegraded":"2024-09-13T16:43:54.997667+0000","last_fullsized":"2024-09-13T16:43:54.997667+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.995002+0000","last_change":"2024-09-13T16:43:54.987328+0000","last_active":"2024-09-13T16:43:54.995002+0000","last_peered":"2024-09-13T16:43:54.995002+0000","last_clean":"2024-09-13T16:43:54.995002+0000","last_became_active":"2024-09-13T16:43:54.987048+0000","last_became_peered":"2024-09-13T16:43:54.987048+0000","last_unstale":"2024-09-13T16:43:54.995002+0000","last_undegraded":"2024-09-13T16:43:54.995002+0000","last_fullsized":"2024-09-13T16:43:54.995002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.979840+0000","last_change":"2024-09-13T16:43:54.985628+0000","last_active":"2024-09-13T16:43:55.979840+0000","last_peered":"2024-09-13T16:43:55.979840+0000","last_clean":"2024-09-13T16:43:55.979840+0000","last_became_active":"2024-09-13T16:43:54.985378+0000","last_became_peered":"2024-09-13T16:43:54.985378+0000","last_unstale":"2024-09-13T16:43:55.979840+0000","last_undegraded":"2024-09-13T16:43:55.979840+0000","last_fullsized":"2024-09-13T16:43:55.979840+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178469+0000","last_change":"2024-09-13T16:43:54.996712+0000","last_active":"2024-09-13T16:43:59.178469+0000","last_peered":"2024-09-13T16:43:59.178469+0000","last_clean":"2024-09-13T16:43:59.178469+0000","last_became_active":"2024-09-13T16:43:54.996523+0000","last_became_peered":"2024-09-13T16:43:54.996523+0000","last_unstale":"2024-09-13T16:43:59.178469+0000","last_undegraded":"2024-09-13T16:43:59.178469+0000","last_fullsized":"2024-09-13T16:43:59.178469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260867+0000","last_change":"2024-09-13T16:43:54.984879+0000","last_active":"2024-09-13T16:43:56.260867+0000","last_peered":"2024-09-13T16:43:56.260867+0000","last_clean":"2024-09-13T16:43:56.260867+0000","last_became_active":"2024-09-13T16:43:54.984595+0000","last_became_peered":"2024-09-13T16:43:54.984595+0000","last_unstale":"2024-09-13T16:43:56.260867+0000","last_undegraded":"2024-09-13T16:43:56.260867+0000","last_fullsized":"2024-09-13T16:43:56.260867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.871914+0000","last_change":"2024-09-13T16:43:54.996457+0000","last_active":"2024-09-13T16:43:56.871914+0000","last_peered":"2024-09-13T16:43:56.871914+0000","last_clean":"2024-09-13T16:43:56.871914+0000","last_became_active":"2024-09-13T16:43:54.996275+0000","last_became_peered":"2024-09-13T16:43:54.996275+0000","last_unstale":"2024-09-13T16:43:56.871914+0000","last_undegraded":"2024-09-13T16:43:56.871914+0000","last_fullsized":"2024-09-13T16:43:56.871914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.993287+0000","last_change":"2024-09-13T16:43:54.985368+0000","last_active":"2024-09-13T16:43:54.993287+0000","last_peered":"2024-09-13T16:43:54.993287+0000","last_clean":"2024-09-13T16:43:54.993287+0000","last_became_active":"2024-09-13T16:43:54.985130+0000","last_became_peered":"2024-09-13T16:43:54.985130+0000","last_unstale":"2024-09-13T16:43:54.993287+0000","last_undegraded":"2024-09-13T16:43:54.993287+0000","last_fullsized":"2024-09-13T16:43:54.993287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508531+0000","last_change":"2024-09-13T16:43:54.991862+0000","last_active":"2024-09-13T16:43:56.508531+0000","last_peered":"2024-09-13T16:43:56.508531+0000","last_clean":"2024-09-13T16:43:56.508531+0000","last_became_active":"2024-09-13T16:43:54.991554+0000","last_became_peered":"2024-09-13T16:43:54.991554+0000","last_unstale":"2024-09-13T16:43:56.508531+0000","last_undegraded":"2024-09-13T16:43:56.508531+0000","last_fullsized":"2024-09-13T16:43:56.508531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.995238+0000","last_change":"2024-09-13T16:43:54.992241+0000","last_active":"2024-09-13T16:43:54.995238+0000","last_peered":"2024-09-13T16:43:54.995238+0000","last_clean":"2024-09-13T16:43:54.995238+0000","last_became_active":"2024-09-13T16:43:54.991687+0000","last_became_peered":"2024-09-13T16:43:54.991687+0000","last_unstale":"2024-09-13T16:43:54.995238+0000","last_undegraded":"2024-09-13T16:43:54.995238+0000","last_fullsized":"2024-09-13T16:43:54.995238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.509024+0000","last_change":"2024-09-13T16:43:54.991366+0000","last_active":"2024-09-13T16:43:56.509024+0000","last_peered":"2024-09-13T16:43:56.509024+0000","last_clean":"2024-09-13T16:43:56.509024+0000","last_became_active":"2024-09-13T16:43:54.991127+0000","last_became_peered":"2024-09-13T16:43:54.991127+0000","last_unstale":"2024-09-13T16:43:56.509024+0000","last_undegraded":"2024-09-13T16:43:56.509024+0000","last_fullsized":"2024-09-13T16:43:56.509024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.261006+0000","last_change":"2024-09-13T16:43:54.985615+0000","last_active":"2024-09-13T16:43:56.261006+0000","last_peered":"2024-09-13T16:43:56.261006+0000","last_clean":"2024-09-13T16:43:56.261006+0000","last_became_active":"2024-09-13T16:43:54.985299+0000","last_became_peered":"2024-09-13T16:43:54.985299+0000","last_unstale":"2024-09-13T16:43:56.261006+0000","last_undegraded":"2024-09-13T16:43:56.261006+0000","last_fullsized":"2024-09-13T16:43:56.261006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975382+0000","last_change":"2024-09-13T16:43:54.993325+0000","last_active":"2024-09-13T16:43:55.975382+0000","last_peered":"2024-09-13T16:43:55.975382+0000","last_clean":"2024-09-13T16:43:55.975382+0000","last_became_active":"2024-09-13T16:43:54.993009+0000","last_became_peered":"2024-09-13T16:43:54.993009+0000","last_unstale":"2024-09-13T16:43:55.975382+0000","last_undegraded":"2024-09-13T16:43:55.975382+0000","last_fullsized":"2024-09-13T16:43:55.975382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508820+0000","last_change":"2024-09-13T16:43:54.998229+0000","last_active":"2024-09-13T16:43:56.508820+0000","last_peered":"2024-09-13T16:43:56.508820+0000","last_clean":"2024-09-13T16:43:56.508820+0000","last_became_active":"2024-09-13T16:43:54.995774+0000","last_became_peered":"2024-09-13T16:43:54.995774+0000","last_unstale":"2024-09-13T16:43:56.508820+0000","last_undegraded":"2024-09-13T16:43:56.508820+0000","last_fullsized":"2024-09-13T16:43:56.508820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872235+0000","last_change":"2024-09-13T16:43:55.002126+0000","last_active":"2024-09-13T16:43:56.872235+0000","last_peered":"2024-09-13T16:43:56.872235+0000","last_clean":"2024-09-13T16:43:56.872235+0000","last_became_active":"2024-09-13T16:43:55.001713+0000","last_became_peered":"2024-09-13T16:43:55.001713+0000","last_unstale":"2024-09-13T16:43:56.872235+0000","last_undegraded":"2024-09-13T16:43:56.872235+0000","last_fullsized":"2024-09-13T16:43:56.872235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.995327+0000","last_change":"2024-09-13T16:43:54.992745+0000","last_active":"2024-09-13T16:43:54.995327+0000","last_peered":"2024-09-13T16:43:54.995327+0000","last_clean":"2024-09-13T16:43:54.995327+0000","last_became_active":"2024-09-13T16:43:54.992508+0000","last_became_peered":"2024-09-13T16:43:54.992508+0000","last_unstale":"2024-09-13T16:43:54.995327+0000","last_undegraded":"2024-09-13T16:43:54.995327+0000","last_fullsized":"2024-09-13T16:43:54.995327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:23.913470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.979793+0000","last_change":"2024-09-13T16:43:54.990631+0000","last_active":"2024-09-13T16:43:55.979793+0000","last_peered":"2024-09-13T16:43:55.979793+0000","last_clean":"2024-09-13T16:43:55.979793+0000","last_became_active":"2024-09-13T16:43:54.986277+0000","last_became_peered":"2024-09-13T16:43:54.986277+0000","last_unstale":"2024-09-13T16:43:55.979793+0000","last_undegraded":"2024-09-13T16:43:55.979793+0000","last_fullsized":"2024-09-13T16:43:55.979793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260674+0000","last_change":"2024-09-13T16:43:55.000059+0000","last_active":"2024-09-13T16:43:56.260674+0000","last_peered":"2024-09-13T16:43:56.260674+0000","last_clean":"2024-09-13T16:43:56.260674+0000","last_became_active":"2024-09-13T16:43:54.999778+0000","last_became_peered":"2024-09-13T16:43:54.999778+0000","last_unstale":"2024-09-13T16:43:56.260674+0000","last_undegraded":"2024-09-13T16:43:56.260674+0000","last_fullsized":"2024-09-13T16:43:56.260674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935551+0000","last_change":"2024-09-13T16:43:54.988375+0000","last_active":"2024-09-13T16:43:56.935551+0000","last_peered":"2024-09-13T16:43:56.935551+0000","last_clean":"2024-09-13T16:43:56.935551+0000","last_became_active":"2024-09-13T16:43:54.988131+0000","last_became_peered":"2024-09-13T16:43:54.988131+0000","last_unstale":"2024-09-13T16:43:56.935551+0000","last_undegraded":"2024-09-13T16:43:56.935551+0000","last_fullsized":"2024-09-13T16:43:56.935551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.635755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.985641+0000","last_change":"2024-09-13T16:43:54.555042+0000","last_active":"2024-09-13T16:43:54.985641+0000","last_peered":"2024-09-13T16:43:54.985641+0000","last_clean":"2024-09-13T16:43:54.985641+0000","last_became_active":"2024-09-13T16:43:54.554697+0000","last_became_peered":"2024-09-13T16:43:54.554697+0000","last_unstale":"2024-09-13T16:43:54.985641+0000","last_undegraded":"2024-09-13T16:43:54.985641+0000","last_fullsized":"2024-09-13T16:43:54.985641+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872075+0000","last_change":"2024-09-13T16:43:54.998755+0000","last_active":"2024-09-13T16:43:56.872075+0000","last_peered":"2024-09-13T16:43:56.872075+0000","last_clean":"2024-09-13T16:43:56.872075+0000","last_became_active":"2024-09-13T16:43:54.998399+0000","last_became_peered":"2024-09-13T16:43:54.998399+0000","last_unstale":"2024-09-13T16:43:56.872075+0000","last_undegraded":"2024-09-13T16:43:56.872075+0000","last_fullsized":"2024-09-13T16:43:56.872075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.978215+0000","last_change":"2024-09-13T16:43:54.553250+0000","last_active":"2024-09-13T16:43:54.978215+0000","last_peered":"2024-09-13T16:43:54.978215+0000","last_clean":"2024-09-13T16:43:54.978215+0000","last_became_active":"2024-09-13T16:43:54.552977+0000","last_became_peered":"2024-09-13T16:43:54.552977+0000","last_unstale":"2024-09-13T16:43:54.978215+0000","last_undegraded":"2024-09-13T16:43:54.978215+0000","last_fullsized":"2024-09-13T16:43:54.978215+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.984357+0000","last_change":"2024-09-13T16:43:54.558065+0000","last_active":"2024-09-13T16:43:54.984357+0000","last_peered":"2024-09-13T16:43:54.984357+0000","last_clean":"2024-09-13T16:43:54.984357+0000","last_became_active":"2024-09-13T16:43:54.557722+0000","last_became_peered":"2024-09-13T16:43:54.557722+0000","last_unstale":"2024-09-13T16:43:54.984357+0000","last_undegraded":"2024-09-13T16:43:54.984357+0000","last_fullsized":"2024-09-13T16:43:54.984357+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508519+0000","last_change":"2024-09-13T16:43:55.002041+0000","last_active":"2024-09-13T16:43:56.508519+0000","last_peered":"2024-09-13T16:43:56.508519+0000","last_clean":"2024-09-13T16:43:56.508519+0000","last_became_active":"2024-09-13T16:43:55.001512+0000","last_became_peered":"2024-09-13T16:43:55.001512+0000","last_unstale":"2024-09-13T16:43:56.508519+0000","last_undegraded":"2024-09-13T16:43:56.508519+0000","last_fullsized":"2024-09-13T16:43:56.508519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.975255+0000","last_change":"2024-09-13T16:43:54.550389+0000","last_active":"2024-09-13T16:43:54.975255+0000","last_peered":"2024-09-13T16:43:54.975255+0000","last_clean":"2024-09-13T16:43:54.975255+0000","last_became_active":"2024-09-13T16:43:54.550154+0000","last_became_peered":"2024-09-13T16:43:54.550154+0000","last_unstale":"2024-09-13T16:43:54.975255+0000","last_undegraded":"2024-09-13T16:43:54.975255+0000","last_fullsized":"2024-09-13T16:43:54.975255+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975641+0000","last_change":"2024-09-13T16:43:54.993377+0000","last_active":"2024-09-13T16:43:55.975641+0000","last_peered":"2024-09-13T16:43:55.975641+0000","last_clean":"2024-09-13T16:43:55.975641+0000","last_became_active":"2024-09-13T16:43:54.993036+0000","last_became_peered":"2024-09-13T16:43:54.993036+0000","last_unstale":"2024-09-13T16:43:55.975641+0000","last_undegraded":"2024-09-13T16:43:55.975641+0000","last_fullsized":"2024-09-13T16:43:55.975641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.978710+0000","last_change":"2024-09-13T16:43:54.555528+0000","last_active":"2024-09-13T16:43:54.978710+0000","last_peered":"2024-09-13T16:43:54.978710+0000","last_clean":"2024-09-13T16:43:54.978710+0000","last_became_active":"2024-09-13T16:43:54.555267+0000","last_became_peered":"2024-09-13T16:43:54.555267+0000","last_unstale":"2024-09-13T16:43:54.978710+0000","last_undegraded":"2024-09-13T16:43:54.978710+0000","last_fullsized":"2024-09-13T16:43:54.978710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.989737+0000","last_change":"2024-09-13T16:43:54.981706+0000","last_active":"2024-09-13T16:43:54.989737+0000","last_peered":"2024-09-13T16:43:54.989737+0000","last_clean":"2024-09-13T16:43:54.989737+0000","last_became_active":"2024-09-13T16:43:54.981429+0000","last_became_peered":"2024-09-13T16:43:54.981429+0000","last_unstale":"2024-09-13T16:43:54.989737+0000","last_undegraded":"2024-09-13T16:43:54.989737+0000","last_fullsized":"2024-09-13T16:43:54.989737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.975723+0000","last_change":"2024-09-13T16:43:54.554593+0000","last_active":"2024-09-13T16:43:54.975723+0000","last_peered":"2024-09-13T16:43:54.975723+0000","last_clean":"2024-09-13T16:43:54.975723+0000","last_became_active":"2024-09-13T16:43:54.554307+0000","last_became_peered":"2024-09-13T16:43:54.554307+0000","last_unstale":"2024-09-13T16:43:54.975723+0000","last_undegraded":"2024-09-13T16:43:54.975723+0000","last_fullsized":"2024-09-13T16:43:54.975723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872139+0000","last_change":"2024-09-13T16:43:54.993485+0000","last_active":"2024-09-13T16:43:56.872139+0000","last_peered":"2024-09-13T16:43:56.872139+0000","last_clean":"2024-09-13T16:43:56.872139+0000","last_became_active":"2024-09-13T16:43:54.993226+0000","last_became_peered":"2024-09-13T16:43:54.993226+0000","last_unstale":"2024-09-13T16:43:56.872139+0000","last_undegraded":"2024-09-13T16:43:56.872139+0000","last_fullsized":"2024-09-13T16:43:56.872139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.975089+0000","last_change":"2024-09-13T16:43:54.562976+0000","last_active":"2024-09-13T16:43:54.975089+0000","last_peered":"2024-09-13T16:43:54.975089+0000","last_clean":"2024-09-13T16:43:54.975089+0000","last_became_active":"2024-09-13T16:43:54.560480+0000","last_became_peered":"2024-09-13T16:43:54.560480+0000","last_unstale":"2024-09-13T16:43:54.975089+0000","last_undegraded":"2024-09-13T16:43:54.975089+0000","last_fullsized":"2024-09-13T16:43:54.975089+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935689+0000","last_change":"2024-09-13T16:43:54.986459+0000","last_active":"2024-09-13T16:43:56.935689+0000","last_peered":"2024-09-13T16:43:56.935689+0000","last_clean":"2024-09-13T16:43:56.935689+0000","last_became_active":"2024-09-13T16:43:54.986083+0000","last_became_peered":"2024-09-13T16:43:54.986083+0000","last_unstale":"2024-09-13T16:43:56.935689+0000","last_undegraded":"2024-09-13T16:43:56.935689+0000","last_fullsized":"2024-09-13T16:43:56.935689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:51.960459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.978381+0000","last_change":"2024-09-13T16:43:54.565929+0000","last_active":"2024-09-13T16:43:54.978381+0000","last_peered":"2024-09-13T16:43:54.978381+0000","last_clean":"2024-09-13T16:43:54.978381+0000","last_became_active":"2024-09-13T16:43:54.565701+0000","last_became_peered":"2024-09-13T16:43:54.565701+0000","last_unstale":"2024-09-13T16:43:54.978381+0000","last_undegraded":"2024-09-13T16:43:54.978381+0000","last_fullsized":"2024-09-13T16:43:54.978381+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178232+0000","last_change":"2024-09-13T16:43:54.989068+0000","last_active":"2024-09-13T16:43:59.178232+0000","last_peered":"2024-09-13T16:43:59.178232+0000","last_clean":"2024-09-13T16:43:59.178232+0000","last_became_active":"2024-09-13T16:43:54.988779+0000","last_became_peered":"2024-09-13T16:43:54.988779+0000","last_unstale":"2024-09-13T16:43:59.178232+0000","last_undegraded":"2024-09-13T16:43:59.178232+0000","last_fullsized":"2024-09-13T16:43:59.178232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.985188+0000","last_change":"2024-09-13T16:43:54.561487+0000","last_active":"2024-09-13T16:43:54.985188+0000","last_peered":"2024-09-13T16:43:54.985188+0000","last_clean":"2024-09-13T16:43:54.985188+0000","last_became_active":"2024-09-13T16:43:54.561077+0000","last_became_peered":"2024-09-13T16:43:54.561077+0000","last_unstale":"2024-09-13T16:43:54.985188+0000","last_undegraded":"2024-09-13T16:43:54.985188+0000","last_fullsized":"2024-09-13T16:43:54.985188+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872141+0000","last_change":"2024-09-13T16:43:54.999517+0000","last_active":"2024-09-13T16:43:56.872141+0000","last_peered":"2024-09-13T16:43:56.872141+0000","last_clean":"2024-09-13T16:43:56.872141+0000","last_became_active":"2024-09-13T16:43:54.999283+0000","last_became_peered":"2024-09-13T16:43:54.999283+0000","last_unstale":"2024-09-13T16:43:56.872141+0000","last_undegraded":"2024-09-13T16:43:56.872141+0000","last_fullsized":"2024-09-13T16:43:56.872141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:20.207775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.975909+0000","last_change":"2024-09-13T16:43:54.542871+0000","last_active":"2024-09-13T16:43:54.975909+0000","last_peered":"2024-09-13T16:43:54.975909+0000","last_clean":"2024-09-13T16:43:54.975909+0000","last_became_active":"2024-09-13T16:43:54.542548+0000","last_became_peered":"2024-09-13T16:43:54.542548+0000","last_unstale":"2024-09-13T16:43:54.975909+0000","last_undegraded":"2024-09-13T16:43:54.975909+0000","last_fullsized":"2024-09-13T16:43:54.975909+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178181+0000","last_change":"2024-09-13T16:43:54.989168+0000","last_active":"2024-09-13T16:43:59.178181+0000","last_peered":"2024-09-13T16:43:59.178181+0000","last_clean":"2024-09-13T16:43:59.178181+0000","last_became_active":"2024-09-13T16:43:54.988716+0000","last_became_peered":"2024-09-13T16:43:54.988716+0000","last_unstale":"2024-09-13T16:43:59.178181+0000","last_undegraded":"2024-09-13T16:43:59.178181+0000","last_fullsized":"2024-09-13T16:43:59.178181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.978600+0000","last_change":"2024-09-13T16:43:54.551551+0000","last_active":"2024-09-13T16:43:54.978600+0000","last_peered":"2024-09-13T16:43:54.978600+0000","last_clean":"2024-09-13T16:43:54.978600+0000","last_became_active":"2024-09-13T16:43:54.551293+0000","last_became_peered":"2024-09-13T16:43:54.551293+0000","last_unstale":"2024-09-13T16:43:54.978600+0000","last_undegraded":"2024-09-13T16:43:54.978600+0000","last_fullsized":"2024-09-13T16:43:54.978600+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260677+0000","last_change":"2024-09-13T16:43:54.994516+0000","last_active":"2024-09-13T16:43:56.260677+0000","last_peered":"2024-09-13T16:43:56.260677+0000","last_clean":"2024-09-13T16:43:56.260677+0000","last_became_active":"2024-09-13T16:43:54.994245+0000","last_became_peered":"2024-09-13T16:43:54.994245+0000","last_unstale":"2024-09-13T16:43:56.260677+0000","last_undegraded":"2024-09-13T16:43:56.260677+0000","last_fullsized":"2024-09-13T16:43:56.260677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.976834+0000","last_change":"2024-09-13T16:43:54.562232+0000","last_active":"2024-09-13T16:43:54.976834+0000","last_peered":"2024-09-13T16:43:54.976834+0000","last_clean":"2024-09-13T16:43:54.976834+0000","last_became_active":"2024-09-13T16:43:54.561939+0000","last_became_peered":"2024-09-13T16:43:54.561939+0000","last_unstale":"2024-09-13T16:43:54.976834+0000","last_undegraded":"2024-09-13T16:43:54.976834+0000","last_fullsized":"2024-09-13T16:43:54.976834+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178047+0000","last_change":"2024-09-13T16:43:54.988439+0000","last_active":"2024-09-13T16:43:59.178047+0000","last_peered":"2024-09-13T16:43:59.178047+0000","last_clean":"2024-09-13T16:43:59.178047+0000","last_became_active":"2024-09-13T16:43:54.988172+0000","last_became_peered":"2024-09-13T16:43:54.988172+0000","last_unstale":"2024-09-13T16:43:59.178047+0000","last_undegraded":"2024-09-13T16:43:59.178047+0000","last_fullsized":"2024-09-13T16:43:59.178047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.978234+0000","last_change":"2024-09-13T16:43:54.559372+0000","last_active":"2024-09-13T16:43:54.978234+0000","last_peered":"2024-09-13T16:43:54.978234+0000","last_clean":"2024-09-13T16:43:54.978234+0000","last_became_active":"2024-09-13T16:43:54.559111+0000","last_became_peered":"2024-09-13T16:43:54.559111+0000","last_unstale":"2024-09-13T16:43:54.978234+0000","last_undegraded":"2024-09-13T16:43:54.978234+0000","last_fullsized":"2024-09-13T16:43:54.978234+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:40.399883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178451+0000","last_change":"2024-09-13T16:43:54.988313+0000","last_active":"2024-09-13T16:43:59.178451+0000","last_peered":"2024-09-13T16:43:59.178451+0000","last_clean":"2024-09-13T16:43:59.178451+0000","last_became_active":"2024-09-13T16:43:54.988059+0000","last_became_peered":"2024-09-13T16:43:54.988059+0000","last_unstale":"2024-09-13T16:43:59.178451+0000","last_undegraded":"2024-09-13T16:43:59.178451+0000","last_fullsized":"2024-09-13T16:43:59.178451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.977092+0000","last_change":"2024-09-13T16:43:54.566410+0000","last_active":"2024-09-13T16:43:54.977092+0000","last_peered":"2024-09-13T16:43:54.977092+0000","last_clean":"2024-09-13T16:43:54.977092+0000","last_became_active":"2024-09-13T16:43:54.565861+0000","last_became_peered":"2024-09-13T16:43:54.565861+0000","last_unstale":"2024-09-13T16:43:54.977092+0000","last_undegraded":"2024-09-13T16:43:54.977092+0000","last_fullsized":"2024-09-13T16:43:54.977092+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178547+0000","last_change":"2024-09-13T16:43:54.989506+0000","last_active":"2024-09-13T16:43:59.178547+0000","last_peered":"2024-09-13T16:43:59.178547+0000","last_clean":"2024-09-13T16:43:59.178547+0000","last_became_active":"2024-09-13T16:43:54.989233+0000","last_became_peered":"2024-09-13T16:43:54.989233+0000","last_unstale":"2024-09-13T16:43:59.178547+0000","last_undegraded":"2024-09-13T16:43:59.178547+0000","last_fullsized":"2024-09-13T16:43:59.178547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:23.913470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.975177+0000","last_change":"2024-09-13T16:43:54.559723+0000","last_active":"2024-09-13T16:43:54.975177+0000","last_peered":"2024-09-13T16:43:54.975177+0000","last_clean":"2024-09-13T16:43:54.975177+0000","last_became_active":"2024-09-13T16:43:54.559353+0000","last_became_peered":"2024-09-13T16:43:54.559353+0000","last_unstale":"2024-09-13T16:43:54.975177+0000","last_undegraded":"2024-09-13T16:43:54.975177+0000","last_fullsized":"2024-09-13T16:43:54.975177+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.995311+0000","last_change":"2024-09-13T16:43:54.992511+0000","last_active":"2024-09-13T16:43:54.995311+0000","last_peered":"2024-09-13T16:43:54.995311+0000","last_clean":"2024-09-13T16:43:54.995311+0000","last_became_active":"2024-09-13T16:43:54.992249+0000","last_became_peered":"2024-09-13T16:43:54.992249+0000","last_unstale":"2024-09-13T16:43:54.995311+0000","last_undegraded":"2024-09-13T16:43:54.995311+0000","last_fullsized":"2024-09-13T16:43:54.995311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.977044+0000","last_change":"2024-09-13T16:43:54.551390+0000","last_active":"2024-09-13T16:43:54.977044+0000","last_peered":"2024-09-13T16:43:54.977044+0000","last_clean":"2024-09-13T16:43:54.977044+0000","last_became_active":"2024-09-13T16:43:54.547074+0000","last_became_peered":"2024-09-13T16:43:54.547074+0000","last_unstale":"2024-09-13T16:43:54.977044+0000","last_undegraded":"2024-09-13T16:43:54.977044+0000","last_fullsized":"2024-09-13T16:43:54.977044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178318+0000","last_change":"2024-09-13T16:43:54.990458+0000","last_active":"2024-09-13T16:43:59.178318+0000","last_peered":"2024-09-13T16:43:59.178318+0000","last_clean":"2024-09-13T16:43:59.178318+0000","last_became_active":"2024-09-13T16:43:54.990196+0000","last_became_peered":"2024-09-13T16:43:54.990196+0000","last_unstale":"2024-09-13T16:43:59.178318+0000","last_undegraded":"2024-09-13T16:43:59.178318+0000","last_fullsized":"2024-09-13T16:43:59.178318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981787+0000","last_change":"2024-09-13T16:43:51.914639+0000","last_active":"2024-09-13T16:43:54.981787+0000","last_peered":"2024-09-13T16:43:54.981787+0000","last_clean":"2024-09-13T16:43:54.981787+0000","last_became_active":"2024-09-13T16:43:51.913797+0000","last_became_peered":"2024-09-13T16:43:51.913797+0000","last_unstale":"2024-09-13T16:43:54.981787+0000","last_undegraded":"2024-09-13T16:43:54.981787+0000","last_fullsized":"2024-09-13T16:43:54.981787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_clean_scrub_stamp":"2024-09-13T16:43:50.498116+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:12.730911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.974632+0000","last_change":"2024-09-13T16:43:54.564153+0000","last_active":"2024-09-13T16:43:54.974632+0000","last_peered":"2024-09-13T16:43:54.974632+0000","last_clean":"2024-09-13T16:43:54.974632+0000","last_became_active":"2024-09-13T16:43:54.563949+0000","last_became_peered":"2024-09-13T16:43:54.563949+0000","last_unstale":"2024-09-13T16:43:54.974632+0000","last_undegraded":"2024-09-13T16:43:54.974632+0000","last_fullsized":"2024-09-13T16:43:54.974632+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508836+0000","last_change":"2024-09-13T16:43:54.997576+0000","last_active":"2024-09-13T16:43:56.508836+0000","last_peered":"2024-09-13T16:43:56.508836+0000","last_clean":"2024-09-13T16:43:56.508836+0000","last_became_active":"2024-09-13T16:43:54.997315+0000","last_became_peered":"2024-09-13T16:43:54.997315+0000","last_unstale":"2024-09-13T16:43:56.508836+0000","last_undegraded":"2024-09-13T16:43:56.508836+0000","last_fullsized":"2024-09-13T16:43:56.508836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:17.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981701+0000","last_change":"2024-09-13T16:43:54.564804+0000","last_active":"2024-09-13T16:43:54.981701+0000","last_peered":"2024-09-13T16:43:54.981701+0000","last_clean":"2024-09-13T16:43:54.981701+0000","last_became_active":"2024-09-13T16:43:54.564605+0000","last_became_peered":"2024-09-13T16:43:54.564605+0000","last_unstale":"2024-09-13T16:43:54.981701+0000","last_undegraded":"2024-09-13T16:43:54.981701+0000","last_fullsized":"2024-09-13T16:43:54.981701+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.871801+0000","last_change":"2024-09-13T16:43:54.997884+0000","last_active":"2024-09-13T16:43:56.871801+0000","last_peered":"2024-09-13T16:43:56.871801+0000","last_clean":"2024-09-13T16:43:56.871801+0000","last_became_active":"2024-09-13T16:43:54.997510+0000","last_became_peered":"2024-09-13T16:43:54.997510+0000","last_unstale":"2024-09-13T16:43:56.871801+0000","last_undegraded":"2024-09-13T16:43:56.871801+0000","last_fullsized":"2024-09-13T16:43:56.871801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.976301+0000","last_change":"2024-09-13T16:43:54.566294+0000","last_active":"2024-09-13T16:43:54.976301+0000","last_peered":"2024-09-13T16:43:54.976301+0000","last_clean":"2024-09-13T16:43:54.976301+0000","last_became_active":"2024-09-13T16:43:54.566023+0000","last_became_peered":"2024-09-13T16:43:54.566023+0000","last_unstale":"2024-09-13T16:43:54.976301+0000","last_undegraded":"2024-09-13T16:43:54.976301+0000","last_fullsized":"2024-09-13T16:43:54.976301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975316+0000","last_change":"2024-09-13T16:43:54.981116+0000","last_active":"2024-09-13T16:43:55.975316+0000","last_peered":"2024-09-13T16:43:55.975316+0000","last_clean":"2024-09-13T16:43:55.975316+0000","last_became_active":"2024-09-13T16:43:54.980766+0000","last_became_peered":"2024-09-13T16:43:54.980766+0000","last_unstale":"2024-09-13T16:43:55.975316+0000","last_undegraded":"2024-09-13T16:43:55.975316+0000","last_fullsized":"2024-09-13T16:43:55.975316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.976007+0000","last_change":"2024-09-13T16:43:54.563716+0000","last_active":"2024-09-13T16:43:54.976007+0000","last_peered":"2024-09-13T16:43:54.976007+0000","last_clean":"2024-09-13T16:43:54.976007+0000","last_became_active":"2024-09-13T16:43:54.563522+0000","last_became_peered":"2024-09-13T16:43:54.563522+0000","last_unstale":"2024-09-13T16:43:54.976007+0000","last_undegraded":"2024-09-13T16:43:54.976007+0000","last_fullsized":"2024-09-13T16:43:54.976007+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935617+0000","last_change":"2024-09-13T16:43:54.999051+0000","last_active":"2024-09-13T16:43:56.935617+0000","last_peered":"2024-09-13T16:43:56.935617+0000","last_clean":"2024-09-13T16:43:56.935617+0000","last_became_active":"2024-09-13T16:43:54.998803+0000","last_became_peered":"2024-09-13T16:43:54.998803+0000","last_unstale":"2024-09-13T16:43:56.935617+0000","last_undegraded":"2024-09-13T16:43:56.935617+0000","last_fullsized":"2024-09-13T16:43:56.935617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.974615+0000","last_change":"2024-09-13T16:43:54.569945+0000","last_active":"2024-09-13T16:43:54.974615+0000","last_peered":"2024-09-13T16:43:54.974615+0000","last_clean":"2024-09-13T16:43:54.974615+0000","last_became_active":"2024-09-13T16:43:54.568846+0000","last_became_peered":"2024-09-13T16:43:54.568846+0000","last_unstale":"2024-09-13T16:43:54.974615+0000","last_undegraded":"2024-09-13T16:43:54.974615+0000","last_fullsized":"2024-09-13T16:43:54.974615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.995256+0000","last_change":"2024-09-13T16:43:54.992082+0000","last_active":"2024-09-13T16:43:54.995256+0000","last_peered":"2024-09-13T16:43:54.995256+0000","last_clean":"2024-09-13T16:43:54.995256+0000","last_became_active":"2024-09-13T16:43:54.991659+0000","last_became_peered":"2024-09-13T16:43:54.991659+0000","last_unstale":"2024-09-13T16:43:54.995256+0000","last_undegraded":"2024-09-13T16:43:54.995256+0000","last_fullsized":"2024-09-13T16:43:54.995256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981345+0000","last_change":"2024-09-13T16:43:54.569111+0000","last_active":"2024-09-13T16:43:54.981345+0000","last_peered":"2024-09-13T16:43:54.981345+0000","last_clean":"2024-09-13T16:43:54.981345+0000","last_became_active":"2024-09-13T16:43:54.568903+0000","last_became_peered":"2024-09-13T16:43:54.568903+0000","last_unstale":"2024-09-13T16:43:54.981345+0000","last_undegraded":"2024-09-13T16:43:54.981345+0000","last_fullsized":"2024-09-13T16:43:54.981345+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975475+0000","last_change":"2024-09-13T16:43:54.980100+0000","last_active":"2024-09-13T16:43:55.975475+0000","last_peered":"2024-09-13T16:43:55.975475+0000","last_clean":"2024-09-13T16:43:55.975475+0000","last_became_active":"2024-09-13T16:43:54.979584+0000","last_became_peered":"2024-09-13T16:43:54.979584+0000","last_unstale":"2024-09-13T16:43:55.975475+0000","last_undegraded":"2024-09-13T16:43:55.975475+0000","last_fullsized":"2024-09-13T16:43:55.975475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.975232+0000","last_change":"2024-09-13T16:43:54.560710+0000","last_active":"2024-09-13T16:43:54.975232+0000","last_peered":"2024-09-13T16:43:54.975232+0000","last_clean":"2024-09-13T16:43:54.975232+0000","last_became_active":"2024-09-13T16:43:54.560030+0000","last_became_peered":"2024-09-13T16:43:54.560030+0000","last_unstale":"2024-09-13T16:43:54.975232+0000","last_undegraded":"2024-09-13T16:43:54.975232+0000","last_fullsized":"2024-09-13T16:43:54.975232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.509048+0000","last_change":"2024-09-13T16:43:54.990812+0000","last_active":"2024-09-13T16:43:56.509048+0000","last_peered":"2024-09-13T16:43:56.509048+0000","last_clean":"2024-09-13T16:43:56.509048+0000","last_became_active":"2024-09-13T16:43:54.990440+0000","last_became_peered":"2024-09-13T16:43:54.990440+0000","last_unstale":"2024-09-13T16:43:56.509048+0000","last_undegraded":"2024-09-13T16:43:56.509048+0000","last_fullsized":"2024-09-13T16:43:56.509048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.978914+0000","last_change":"2024-09-13T16:43:54.558439+0000","last_active":"2024-09-13T16:43:54.978914+0000","last_peered":"2024-09-13T16:43:54.978914+0000","last_clean":"2024-09-13T16:43:54.978914+0000","last_became_active":"2024-09-13T16:43:54.558194+0000","last_became_peered":"2024-09-13T16:43:54.558194+0000","last_unstale":"2024-09-13T16:43:54.978914+0000","last_undegraded":"2024-09-13T16:43:54.978914+0000","last_fullsized":"2024-09-13T16:43:54.978914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872087+0000","last_change":"2024-09-13T16:43:54.995534+0000","last_active":"2024-09-13T16:43:56.872087+0000","last_peered":"2024-09-13T16:43:56.872087+0000","last_clean":"2024-09-13T16:43:56.872087+0000","last_became_active":"2024-09-13T16:43:54.995397+0000","last_became_peered":"2024-09-13T16:43:54.995397+0000","last_unstale":"2024-09-13T16:43:56.872087+0000","last_undegraded":"2024-09-13T16:43:56.872087+0000","last_fullsized":"2024-09-13T16:43:56.872087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:19.697247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981614+0000","last_change":"2024-09-13T16:43:54.569085+0000","last_active":"2024-09-13T16:43:54.981614+0000","last_peered":"2024-09-13T16:43:54.981614+0000","last_clean":"2024-09-13T16:43:54.981614+0000","last_became_active":"2024-09-13T16:43:54.568892+0000","last_became_peered":"2024-09-13T16:43:54.568892+0000","last_unstale":"2024-09-13T16:43:54.981614+0000","last_undegraded":"2024-09-13T16:43:54.981614+0000","last_fullsized":"2024-09-13T16:43:54.981614+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935459+0000","last_change":"2024-09-13T16:43:54.984120+0000","last_active":"2024-09-13T16:43:56.935459+0000","last_peered":"2024-09-13T16:43:56.935459+0000","last_clean":"2024-09-13T16:43:56.935459+0000","last_became_active":"2024-09-13T16:43:54.983886+0000","last_became_peered":"2024-09-13T16:43:54.983886+0000","last_unstale":"2024-09-13T16:43:56.935459+0000","last_undegraded":"2024-09-13T16:43:56.935459+0000","last_fullsized":"2024-09-13T16:43:56.935459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.973946+0000","last_change":"2024-09-13T16:43:54.555092+0000","last_active":"2024-09-13T16:43:54.973946+0000","last_peered":"2024-09-13T16:43:54.973946+0000","last_clean":"2024-09-13T16:43:54.973946+0000","last_became_active":"2024-09-13T16:43:54.554230+0000","last_became_peered":"2024-09-13T16:43:54.554230+0000","last_unstale":"2024-09-13T16:43:54.973946+0000","last_undegraded":"2024-09-13T16:43:54.973946+0000","last_fullsized":"2024-09-13T16:43:54.973946+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508681+0000","last_change":"2024-09-13T16:43:54.984588+0000","last_active":"2024-09-13T16:43:56.508681+0000","last_peered":"2024-09-13T16:43:56.508681+0000","last_clean":"2024-09-13T16:43:56.508681+0000","last_became_active":"2024-09-13T16:43:54.984221+0000","last_became_peered":"2024-09-13T16:43:54.984221+0000","last_unstale":"2024-09-13T16:43:56.508681+0000","last_undegraded":"2024-09-13T16:43:56.508681+0000","last_fullsized":"2024-09-13T16:43:56.508681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.973687+0000","last_change":"2024-09-13T16:43:54.562646+0000","last_active":"2024-09-13T16:43:54.973687+0000","last_peered":"2024-09-13T16:43:54.973687+0000","last_clean":"2024-09-13T16:43:54.973687+0000","last_became_active":"2024-09-13T16:43:54.556888+0000","last_became_peered":"2024-09-13T16:43:54.556888+0000","last_unstale":"2024-09-13T16:43:54.973687+0000","last_undegraded":"2024-09-13T16:43:54.973687+0000","last_fullsized":"2024-09-13T16:43:54.973687+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508792+0000","last_change":"2024-09-13T16:43:54.999328+0000","last_active":"2024-09-13T16:43:56.508792+0000","last_peered":"2024-09-13T16:43:56.508792+0000","last_clean":"2024-09-13T16:43:56.508792+0000","last_became_active":"2024-09-13T16:43:54.999101+0000","last_became_peered":"2024-09-13T16:43:54.999101+0000","last_unstale":"2024-09-13T16:43:56.508792+0000","last_undegraded":"2024-09-13T16:43:56.508792+0000","last_fullsized":"2024-09-13T16:43:56.508792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.979629+0000","last_change":"2024-09-13T16:43:54.559214+0000","last_active":"2024-09-13T16:43:54.979629+0000","last_peered":"2024-09-13T16:43:54.979629+0000","last_clean":"2024-09-13T16:43:54.979629+0000","last_became_active":"2024-09-13T16:43:54.558233+0000","last_became_peered":"2024-09-13T16:43:54.558233+0000","last_unstale":"2024-09-13T16:43:54.979629+0000","last_undegraded":"2024-09-13T16:43:54.979629+0000","last_fullsized":"2024-09-13T16:43:54.979629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260789+0000","last_change":"2024-09-13T16:43:54.986565+0000","last_active":"2024-09-13T16:43:56.260789+0000","last_peered":"2024-09-13T16:43:56.260789+0000","last_clean":"2024-09-13T16:43:56.260789+0000","last_became_active":"2024-09-13T16:43:54.986084+0000","last_became_peered":"2024-09-13T16:43:54.986084+0000","last_unstale":"2024-09-13T16:43:56.260789+0000","last_undegraded":"2024-09-13T16:43:56.260789+0000","last_fullsized":"2024-09-13T16:43:56.260789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.989817+0000","last_change":"2024-09-13T16:43:54.982611+0000","last_active":"2024-09-13T16:43:54.989817+0000","last_peered":"2024-09-13T16:43:54.989817+0000","last_clean":"2024-09-13T16:43:54.989817+0000","last_became_active":"2024-09-13T16:43:54.982338+0000","last_became_peered":"2024-09-13T16:43:54.982338+0000","last_unstale":"2024-09-13T16:43:54.989817+0000","last_undegraded":"2024-09-13T16:43:54.989817+0000","last_fullsized":"2024-09-13T16:43:54.989817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.979247+0000","last_change":"2024-09-13T16:43:54.558799+0000","last_active":"2024-09-13T16:43:54.979247+0000","last_peered":"2024-09-13T16:43:54.979247+0000","last_clean":"2024-09-13T16:43:54.979247+0000","last_became_active":"2024-09-13T16:43:54.558332+0000","last_became_peered":"2024-09-13T16:43:54.558332+0000","last_unstale":"2024-09-13T16:43:54.979247+0000","last_undegraded":"2024-09-13T16:43:54.979247+0000","last_fullsized":"2024-09-13T16:43:54.979247+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:19.201506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.993116+0000","last_change":"2024-09-13T16:43:54.984150+0000","last_active":"2024-09-13T16:43:54.993116+0000","last_peered":"2024-09-13T16:43:54.993116+0000","last_clean":"2024-09-13T16:43:54.993116+0000","last_became_active":"2024-09-13T16:43:54.983881+0000","last_became_peered":"2024-09-13T16:43:54.983881+0000","last_unstale":"2024-09-13T16:43:54.993116+0000","last_undegraded":"2024-09-13T16:43:54.993116+0000","last_fullsized":"2024-09-13T16:43:54.993116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:29.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.980937+0000","last_change":"2024-09-13T16:43:54.559551+0000","last_active":"2024-09-13T16:43:54.980937+0000","last_peered":"2024-09-13T16:43:54.980937+0000","last_clean":"2024-09-13T16:43:54.980937+0000","last_became_active":"2024-09-13T16:43:54.559290+0000","last_became_peered":"2024-09-13T16:43:54.559290+0000","last_unstale":"2024-09-13T16:43:54.980937+0000","last_undegraded":"2024-09-13T16:43:54.980937+0000","last_fullsized":"2024-09-13T16:43:54.980937+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872084+0000","last_change":"2024-09-13T16:43:55.000783+0000","last_active":"2024-09-13T16:43:56.872084+0000","last_peered":"2024-09-13T16:43:56.872084+0000","last_clean":"2024-09-13T16:43:56.872084+0000","last_became_active":"2024-09-13T16:43:55.000196+0000","last_became_peered":"2024-09-13T16:43:55.000196+0000","last_unstale":"2024-09-13T16:43:56.872084+0000","last_undegraded":"2024-09-13T16:43:56.872084+0000","last_fullsized":"2024-09-13T16:43:56.872084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.986648+0000","last_change":"2024-09-13T16:43:54.559886+0000","last_active":"2024-09-13T16:43:54.986648+0000","last_peered":"2024-09-13T16:43:54.986648+0000","last_clean":"2024-09-13T16:43:54.986648+0000","last_became_active":"2024-09-13T16:43:54.559586+0000","last_became_peered":"2024-09-13T16:43:54.559586+0000","last_unstale":"2024-09-13T16:43:54.986648+0000","last_undegraded":"2024-09-13T16:43:54.986648+0000","last_fullsized":"2024-09-13T16:43:54.986648+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508978+0000","last_change":"2024-09-13T16:43:54.996099+0000","last_active":"2024-09-13T16:43:56.508978+0000","last_peered":"2024-09-13T16:43:56.508978+0000","last_clean":"2024-09-13T16:43:56.508978+0000","last_became_active":"2024-09-13T16:43:54.995802+0000","last_became_peered":"2024-09-13T16:43:54.995802+0000","last_unstale":"2024-09-13T16:43:56.508978+0000","last_undegraded":"2024-09-13T16:43:56.508978+0000","last_fullsized":"2024-09-13T16:43:56.508978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.983458+0000","last_change":"2024-09-13T16:43:54.562699+0000","last_active":"2024-09-13T16:43:54.983458+0000","last_peered":"2024-09-13T16:43:54.983458+0000","last_clean":"2024-09-13T16:43:54.983458+0000","last_became_active":"2024-09-13T16:43:54.560552+0000","last_became_peered":"2024-09-13T16:43:54.560552+0000","last_unstale":"2024-09-13T16:43:54.983458+0000","last_undegraded":"2024-09-13T16:43:54.983458+0000","last_fullsized":"2024-09-13T16:43:54.983458+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:11.032382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":654966,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574850,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":176128,"data_stored":57209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":70284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":266240,"data_stored":660247,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:44:01.574 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T16:44:01.575 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T16:44:01.575 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-13T16:44:01.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-13T16:44:01.575 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-13T16:44:01.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.764+0000 7f4994e00640 1 Processor -- start 2024-09-13T16:44:01.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.764+0000 7f4994e00640 1 -- start start 2024-09-13T16:44:01.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.765+0000 7f4994e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f49901509b0 0x7f499006cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:01.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.765+0000 7f4994e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f49900c6240 0x7f499006d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:01.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.765+0000 7f4994e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499006e100 con 0x7f49900c1b40 2024-09-13T16:44:01.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.765+0000 7f4994e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4990150d90 con 0x7f49900c6240 2024-09-13T16:44:01.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.765+0000 7f4994e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49900c43b0 con 0x7f49901509b0 2024-09-13T16:44:01.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.765+0000 7f498fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f49900c6240 0x7f499006d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:01.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.765+0000 7f498fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f49900c6240 0x7f499006d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41378/0 (socket says 172.21.15.28:41378) 2024-09-13T16:44:01.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.765+0000 7f498fe00640 1 -- 172.21.15.28:0/1183030201 learned_addr learned my addr 172.21.15.28:0/1183030201 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:01.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.765+0000 7f498f400640 1 --2- 172.21.15.28:0/1183030201 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f49901509b0 0x7f499006cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:01.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.766+0000 7f498fe00640 1 -- 172.21.15.28:0/1183030201 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f49901509b0 msgr2=0x7f499006cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:01.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.766+0000 7f498fe00640 1 --2- 172.21.15.28:0/1183030201 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f49901509b0 0x7f499006cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.766+0000 7f498fe00640 1 -- 172.21.15.28:0/1183030201 >> v1:172.21.15.28:6789/0 conn(0x7f49900c1b40 legacy=0x7f4990156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:01.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.766+0000 7f498fe00640 1 -- 172.21.15.28:0/1183030201 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499006d7d0 con 0x7f49900c6240 2024-09-13T16:44:01.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.766+0000 7f498ea00640 1 -- 172.21.15.28:0/1183030201 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487399068 0 0) 0x7f499006e100 con 0x7f49900c1b40 2024-09-13T16:44:01.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.767+0000 7f498fe00640 1 --2- 172.21.15.28:0/1183030201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49900c6240 0x7f499006d0e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4988009e50 tx=0x7f498802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688ea9e3ba2e39c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:01.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.767+0000 7f498ea00640 1 -- 172.21.15.28:0/1183030201 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4988031600 con 0x7f49900c6240 2024-09-13T16:44:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.767+0000 7f498ea00640 1 -- 172.21.15.28:0/1183030201 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498802d050 con 0x7f49900c6240 2024-09-13T16:44:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.767+0000 7f498ea00640 1 -- 172.21.15.28:0/1183030201 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4988031980 con 0x7f49900c6240 2024-09-13T16:44:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.767+0000 7f4994e00640 1 -- 172.21.15.28:0/1183030201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49900c6240 msgr2=0x7f499006d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.768+0000 7f4994e00640 1 --2- 172.21.15.28:0/1183030201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49900c6240 0x7f499006d0e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4988009e50 tx=0x7f498802a710 comp rx=0 tx=0).stop 2024-09-13T16:44:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.768+0000 7f4994e00640 1 -- 172.21.15.28:0/1183030201 shutdown_connections 2024-09-13T16:44:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.768+0000 7f4994e00640 1 --2- 172.21.15.28:0/1183030201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49900c6240 0x7f499006d0e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.768+0000 7f4994e00640 1 --2- 172.21.15.28:0/1183030201 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f49901509b0 0x7f499006cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.768+0000 7f4994e00640 1 -- 172.21.15.28:0/1183030201 >> 172.21.15.28:0/1183030201 conn(0x7f49900ca0e0 msgr2=0x7f49900ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.768+0000 7f4994e00640 1 -- 172.21.15.28:0/1183030201 shutdown_connections 2024-09-13T16:44:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.768+0000 7f4994e00640 1 -- 172.21.15.28:0/1183030201 wait complete. 2024-09-13T16:44:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.769+0000 7f4994e00640 1 Processor -- start 2024-09-13T16:44:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.769+0000 7f4994e00640 1 -- start start 2024-09-13T16:44:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.770+0000 7f4994e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49900c1b40 0x7f49901542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.770+0000 7f4994e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f49900c6240 0x7f4990146cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.770+0000 7f4994e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49901509b0 0x7f49901ee3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.770+0000 7f4994e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49901e3ce0 con 0x7f49901509b0 2024-09-13T16:44:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.770+0000 7f4994e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4990155fc0 con 0x7f49900c1b40 2024-09-13T16:44:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.770+0000 7f4994e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4990156280 con 0x7f49900c6240 2024-09-13T16:44:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.770+0000 7f498fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49901509b0 0x7f49901ee3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.770+0000 7f498fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49901509b0 0x7f49901ee3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41386/0 (socket says 172.21.15.28:41386) 2024-09-13T16:44:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.770+0000 7f498fe00640 1 -- 172.21.15.28:0/2506173841 learned_addr learned my addr 172.21.15.28:0/2506173841 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.771+0000 7f498f400640 1 --2- 172.21.15.28:0/2506173841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49900c1b40 0x7f49901542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.771+0000 7f4986a00640 1 --2- 172.21.15.28:0/2506173841 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f49900c6240 0x7f4990146cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.771+0000 7f498fe00640 1 -- 172.21.15.28:0/2506173841 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f49900c6240 msgr2=0x7f4990146cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.771+0000 7f498fe00640 1 --2- 172.21.15.28:0/2506173841 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f49900c6240 0x7f4990146cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.771+0000 7f498fe00640 1 -- 172.21.15.28:0/2506173841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49900c1b40 msgr2=0x7f49901542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.771+0000 7f498fe00640 1 --2- 172.21.15.28:0/2506173841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49900c1b40 0x7f49901542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.771+0000 7f498fe00640 1 -- 172.21.15.28:0/2506173841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4988009b30 con 0x7f49901509b0 2024-09-13T16:44:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.771+0000 7f498fe00640 1 --2- 172.21.15.28:0/2506173841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49901509b0 0x7f49901ee3f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4988034040 tx=0x7f4988004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.771+0000 7f4987e00640 1 -- 172.21.15.28:0/2506173841 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498803d860 con 0x7f49901509b0 2024-09-13T16:44:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.771+0000 7f4987e00640 1 -- 172.21.15.28:0/2506173841 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498803d9e0 con 0x7f49901509b0 2024-09-13T16:44:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.771+0000 7f4987e00640 1 -- 172.21.15.28:0/2506173841 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498803dd90 con 0x7f49901509b0 2024-09-13T16:44:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.772+0000 7f4994e00640 1 -- 172.21.15.28:0/2506173841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49901e3200 con 0x7f49901509b0 2024-09-13T16:44:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.772+0000 7f4994e00640 1 -- 172.21.15.28:0/2506173841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49901e37c0 con 0x7f49901509b0 2024-09-13T16:44:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.773+0000 7f4994e00640 1 -- 172.21.15.28:0/2506173841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49900c3d80 con 0x7f49901509b0 2024-09-13T16:44:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.773+0000 7f4987e00640 1 -- 172.21.15.28:0/2506173841 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f498803b400 con 0x7f49901509b0 2024-09-13T16:44:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.774+0000 7f4987e00640 1 --2- 172.21.15.28:0/2506173841 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f496c06ac30 0x7f496c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.774+0000 7f4987e00640 1 -- 172.21.15.28:0/2506173841 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49880aa1b0 con 0x7f49901509b0 2024-09-13T16:44:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.775+0000 7f498f400640 1 --2- 172.21.15.28:0/2506173841 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f496c06ac30 0x7f496c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:01.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.775+0000 7f498f400640 1 --2- 172.21.15.28:0/2506173841 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f496c06ac30 0x7f496c06d0f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4980009aa0 tx=0x7f4980009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:01.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:01.778+0000 7f4987e00640 1 -- 172.21.15.28:0/2506173841 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4988075310 con 0x7f49901509b0 2024-09-13T16:44:02.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.004+0000 7f4994e00640 1 -- 172.21.15.28:0/2506173841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f49901e4800 con 0x7f49901509b0 2024-09-13T16:44:02.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.007+0000 7f4987e00640 1 -- 172.21.15.28:0/2506173841 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170959 (secure 0 0 0) 0x7f4988009510 con 0x7f49901509b0 2024-09-13T16:44:02.011 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:02.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.010+0000 7f4994e00640 1 -- 172.21.15.28:0/2506173841 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f496c06ac30 msgr2=0x7f496c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:02.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.010+0000 7f4994e00640 1 --2- 172.21.15.28:0/2506173841 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f496c06ac30 0x7f496c06d0f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4980009aa0 tx=0x7f4980009190 comp rx=0 tx=0).stop 2024-09-13T16:44:02.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.010+0000 7f4994e00640 1 -- 172.21.15.28:0/2506173841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49901509b0 msgr2=0x7f49901ee3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:02.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.010+0000 7f4994e00640 1 --2- 172.21.15.28:0/2506173841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49901509b0 0x7f49901ee3f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4988034040 tx=0x7f4988004a40 comp rx=0 tx=0).stop 2024-09-13T16:44:02.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.010+0000 7f4994e00640 1 -- 172.21.15.28:0/2506173841 shutdown_connections 2024-09-13T16:44:02.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.010+0000 7f4994e00640 1 --2- 172.21.15.28:0/2506173841 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f496c06ac30 0x7f496c06d0f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.010+0000 7f4994e00640 1 --2- 172.21.15.28:0/2506173841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49901509b0 0x7f49901ee3f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.010+0000 7f4994e00640 1 --2- 172.21.15.28:0/2506173841 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f49900c6240 0x7f4990146cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.010+0000 7f4994e00640 1 --2- 172.21.15.28:0/2506173841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49900c1b40 0x7f49901542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.010+0000 7f4994e00640 1 -- 172.21.15.28:0/2506173841 >> 172.21.15.28:0/2506173841 conn(0x7f49900ca0e0 msgr2=0x7f49900c47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:02.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.011+0000 7f4994e00640 1 -- 172.21.15.28:0/2506173841 shutdown_connections 2024-09-13T16:44:02.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.011+0000 7f4994e00640 1 -- 172.21.15.28:0/2506173841 wait complete. 2024-09-13T16:44:02.028 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"flags":0,"active_gid":4109,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":1568228941},{"type":"v1","addr":"172.21.15.191:6833","nonce":1568228941}]},"active_addr":"172.21.15.191:6833/1568228941","active_change":"2024-09-13T16:43:43.580347+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4115,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.191:0","nonce":1482473171}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.191:0","nonce":1253631577}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.191:0","nonce":2125347688}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.191:0","nonce":2927749127}]}]} 2024-09-13T16:44:02.034 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-13T16:44:02.034 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T16:44:02.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:44:02.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.186+0000 7f8ff1200640 1 Processor -- start 2024-09-13T16:44:02.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.186+0000 7f8ff1200640 1 -- start start 2024-09-13T16:44:02.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.188+0000 7f8ff1200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8fec14f790 0x7f8fec0cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:02.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.188+0000 7f8ff1200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fec14fb70 0x7f8fec0c3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:02.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.188+0000 7f8ff1200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8fec0c42f0 0x7f8fec0c3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:02.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.188+0000 7f8ff1200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fec0c1c80 con 0x7f8fec14f790 2024-09-13T16:44:02.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.188+0000 7f8ff1200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fec06cb60 con 0x7f8fec0c42f0 2024-09-13T16:44:02.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.188+0000 7f8ff1200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec06ce40 con 0x7f8fec14fb70 2024-09-13T16:44:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.188+0000 7f8feb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8fec14f790 0x7f8fec0cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.188+0000 7f8feb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8fec14f790 0x7f8fec0cabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41402/0 (socket says 172.21.15.28:41402) 2024-09-13T16:44:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.188+0000 7f8feb400640 1 -- 172.21.15.28:0/1693578525 learned_addr learned my addr 172.21.15.28:0/1693578525 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.188+0000 7f8febe00640 1 --2- 172.21.15.28:0/1693578525 >> v2:172.21.15.191:3300/0 conn(0x7f8fec0c42f0 0x7f8fec0c3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.188+0000 7f8fe2a00640 1 --2- 172.21.15.28:0/1693578525 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fec14fb70 0x7f8fec0c3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.188+0000 7f8feb400640 1 -- 172.21.15.28:0/1693578525 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fec14fb70 msgr2=0x7f8fec0c3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.188+0000 7f8feb400640 1 --2- 172.21.15.28:0/1693578525 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fec14fb70 0x7f8fec0c3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.188+0000 7f8feb400640 1 -- 172.21.15.28:0/1693578525 >> v2:172.21.15.191:3300/0 conn(0x7f8fec0c42f0 msgr2=0x7f8fec0c3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.188+0000 7f8feb400640 1 --2- 172.21.15.28:0/1693578525 >> v2:172.21.15.191:3300/0 conn(0x7f8fec0c42f0 0x7f8fec0c3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.188+0000 7f8feb400640 1 -- 172.21.15.28:0/1693578525 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec156070 con 0x7f8fec14f790 2024-09-13T16:44:02.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.189+0000 7f8feb400640 1 --2- 172.21.15.28:0/1693578525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec14f790 0x7f8fec0cabd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0c7f70 tx=0x7f8fdc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=231b45a81a87c1df server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:02.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.190+0000 7f8feaa00640 1 -- 172.21.15.28:0/1693578525 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc032440 con 0x7f8fec14f790 2024-09-13T16:44:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.190+0000 7f8feaa00640 1 -- 172.21.15.28:0/1693578525 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc0325a0 con 0x7f8fec14f790 2024-09-13T16:44:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.190+0000 7f8feaa00640 1 -- 172.21.15.28:0/1693578525 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc032930 con 0x7f8fec14f790 2024-09-13T16:44:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.190+0000 7f8ff1200640 1 -- 172.21.15.28:0/1693578525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec14f790 msgr2=0x7f8fec0cabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.190+0000 7f8ff1200640 1 --2- 172.21.15.28:0/1693578525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec14f790 0x7f8fec0cabd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0c7f70 tx=0x7f8fdc02a270 comp rx=0 tx=0).stop 2024-09-13T16:44:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.190+0000 7f8ff1200640 1 -- 172.21.15.28:0/1693578525 shutdown_connections 2024-09-13T16:44:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.190+0000 7f8ff1200640 1 --2- 172.21.15.28:0/1693578525 >> v2:172.21.15.191:3300/0 conn(0x7f8fec0c42f0 0x7f8fec0c3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.190+0000 7f8ff1200640 1 --2- 172.21.15.28:0/1693578525 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fec14fb70 0x7f8fec0c3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.190+0000 7f8ff1200640 1 --2- 172.21.15.28:0/1693578525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec14f790 0x7f8fec0cabd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.190+0000 7f8ff1200640 1 -- 172.21.15.28:0/1693578525 >> 172.21.15.28:0/1693578525 conn(0x7f8fec0ca0e0 msgr2=0x7f8fec0ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:02.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.190+0000 7f8ff1200640 1 -- 172.21.15.28:0/1693578525 shutdown_connections 2024-09-13T16:44:02.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.190+0000 7f8ff1200640 1 -- 172.21.15.28:0/1693578525 wait complete. 2024-09-13T16:44:02.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.192+0000 7f8ff1200640 1 Processor -- start 2024-09-13T16:44:02.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.192+0000 7f8ff1200640 1 -- start start 2024-09-13T16:44:02.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.193+0000 7f8ff1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec0c42f0 0x7f8fec1520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:02.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.193+0000 7f8ff1200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fec14f790 0x7f8fec1e2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:02.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.193+0000 7f8ff1200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fec1ef410 0x7f8fec1ef8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:02.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.193+0000 7f8ff1200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec1517d0 con 0x7f8fec0c42f0 2024-09-13T16:44:02.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.193+0000 7f8ff1200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec1525f0 con 0x7f8fec14f790 2024-09-13T16:44:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.193+0000 7f8ff1200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec1e3750 con 0x7f8fec1ef410 2024-09-13T16:44:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.193+0000 7f8feb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec0c42f0 0x7f8fec1520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.193+0000 7f8feb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec0c42f0 0x7f8fec1520b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41404/0 (socket says 172.21.15.28:41404) 2024-09-13T16:44:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.193+0000 7f8feb400640 1 -- 172.21.15.28:0/3819080988 learned_addr learned my addr 172.21.15.28:0/3819080988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.194+0000 7f8fe2a00640 1 --2- 172.21.15.28:0/3819080988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fec14f790 0x7f8fec1e2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.194+0000 7f8febe00640 1 --2- 172.21.15.28:0/3819080988 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fec1ef410 0x7f8fec1ef8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.194+0000 7f8feb400640 1 -- 172.21.15.28:0/3819080988 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fec1ef410 msgr2=0x7f8fec1ef8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.194+0000 7f8feb400640 1 --2- 172.21.15.28:0/3819080988 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fec1ef410 0x7f8fec1ef8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.194+0000 7f8feb400640 1 -- 172.21.15.28:0/3819080988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fec14f790 msgr2=0x7f8fec1e2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.194+0000 7f8feb400640 1 --2- 172.21.15.28:0/3819080988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fec14f790 0x7f8fec1e2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.194+0000 7f8feb400640 1 -- 172.21.15.28:0/3819080988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fdc0094d0 con 0x7f8fec0c42f0 2024-09-13T16:44:02.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.194+0000 7f8feb400640 1 --2- 172.21.15.28:0/3819080988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec0c42f0 0x7f8fec1520b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc02a2b0 tx=0x7f8fdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:02.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.194+0000 7f8fe3e00640 1 -- 172.21.15.28:0/3819080988 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc03b810 con 0x7f8fec0c42f0 2024-09-13T16:44:02.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.194+0000 7f8fe3e00640 1 -- 172.21.15.28:0/3819080988 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc03b970 con 0x7f8fec0c42f0 2024-09-13T16:44:02.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.194+0000 7f8fe3e00640 1 -- 172.21.15.28:0/3819080988 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc03bd00 con 0x7f8fec0c42f0 2024-09-13T16:44:02.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.194+0000 7f8ff1200640 1 -- 172.21.15.28:0/3819080988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fec1e39f0 con 0x7f8fec0c42f0 2024-09-13T16:44:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.194+0000 7f8ff1200640 1 -- 172.21.15.28:0/3819080988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fec150cf0 con 0x7f8fec0c42f0 2024-09-13T16:44:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.195+0000 7f8ff1200640 1 -- 172.21.15.28:0/3819080988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fec0c9620 con 0x7f8fec0c42f0 2024-09-13T16:44:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.196+0000 7f8fe3e00640 1 -- 172.21.15.28:0/3819080988 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8fdc039040 con 0x7f8fec0c42f0 2024-09-13T16:44:02.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.197+0000 7f8fe3e00640 1 --2- 172.21.15.28:0/3819080988 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8fc806ac00 0x7f8fc806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:02.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.197+0000 7f8fe3e00640 1 -- 172.21.15.28:0/3819080988 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8fdc0a9980 con 0x7f8fec0c42f0 2024-09-13T16:44:02.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.197+0000 7f8fe2a00640 1 --2- 172.21.15.28:0/3819080988 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8fc806ac00 0x7f8fc806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:02.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.198+0000 7f8fe2a00640 1 --2- 172.21.15.28:0/3819080988 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8fc806ac00 0x7f8fc806d0c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0c2a10 tx=0x7f8fd4009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:02.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.201+0000 7f8fe3e00640 1 -- 172.21.15.28:0/3819080988 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8fdc074d20 con 0x7f8fec0c42f0 2024-09-13T16:44:02.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.379+0000 7f8ff1200640 1 -- 172.21.15.28:0/3819080988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fec0cabd0 con 0x7f8fec0c42f0 2024-09-13T16:44:02.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.380+0000 7f8fe3e00640 1 -- 172.21.15.28:0/3819080988 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16481 (secure 0 0 0) 0x7f8fdc037090 con 0x7f8fec0c42f0 2024-09-13T16:44:02.382 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:02.382 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:43:55.971358+0000","last_up_change":"2024-09-13T16:43:49.476313+0000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:43:49.643606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:43:52.828510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:43:53.945874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6801","nonce":2964323435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6803","nonce":2964323435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6807","nonce":2964323435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6805","nonce":2964323435}]},"public_addr":"172.21.15.28:6801/2964323435","cluster_addr":"172.21.15.28:6803/2964323435","heartbeat_back_addr":"172.21.15.28:6807/2964323435","heartbeat_front_addr":"172.21.15.28:6805/2964323435","state":["exists","up"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6809","nonce":23768570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6811","nonce":23768570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6815","nonce":23768570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6813","nonce":23768570}]},"public_addr":"172.21.15.28:6809/23768570","cluster_addr":"172.21.15.28:6811/23768570","heartbeat_back_addr":"172.21.15.28:6815/23768570","heartbeat_front_addr":"172.21.15.28:6813/23768570","state":["exists","up"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6817","nonce":886887303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6819","nonce":886887303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6823","nonce":886887303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6821","nonce":886887303}]},"public_addr":"172.21.15.28:6817/886887303","cluster_addr":"172.21.15.28:6819/886887303","heartbeat_back_addr":"172.21.15.28:6823/886887303","heartbeat_front_addr":"172.21.15.28:6821/886887303","state":["exists","up"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6825","nonce":2616036674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6827","nonce":2616036674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6831","nonce":2616036674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6829","nonce":2616036674}]},"public_addr":"172.21.15.28:6825/2616036674","cluster_addr":"172.21.15.28:6827/2616036674","heartbeat_back_addr":"172.21.15.28:6831/2616036674","heartbeat_front_addr":"172.21.15.28:6829/2616036674","state":["exists","up"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6801","nonce":2817102450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6803","nonce":2817102450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6807","nonce":2817102450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6805","nonce":2817102450}]},"public_addr":"172.21.15.191:6801/2817102450","cluster_addr":"172.21.15.191:6803/2817102450","heartbeat_back_addr":"172.21.15.191:6807/2817102450","heartbeat_front_addr":"172.21.15.191:6805/2817102450","state":["exists","up"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6817","nonce":2503027971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6819","nonce":2503027971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6823","nonce":2503027971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6821","nonce":2503027971}]},"public_addr":"172.21.15.191:6817/2503027971","cluster_addr":"172.21.15.191:6819/2503027971","heartbeat_back_addr":"172.21.15.191:6823/2503027971","heartbeat_front_addr":"172.21.15.191:6821/2503027971","state":["exists","up"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6809","nonce":3006595088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6811","nonce":3006595088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6815","nonce":3006595088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6813","nonce":3006595088}]},"public_addr":"172.21.15.191:6809/3006595088","cluster_addr":"172.21.15.191:6811/3006595088","heartbeat_back_addr":"172.21.15.191:6815/3006595088","heartbeat_front_addr":"172.21.15.191:6813/3006595088","state":["exists","up"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6825","nonce":1477327720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6827","nonce":1477327720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6831","nonce":1477327720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6829","nonce":1477327720}]},"public_addr":"172.21.15.191:6825/1477327720","cluster_addr":"172.21.15.191:6827/1477327720","heartbeat_back_addr":"172.21.15.191:6831/1477327720","heartbeat_front_addr":"172.21.15.191:6829/1477327720","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.917250+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.954351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.017329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.046221+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.707467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.726947+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.782321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.764850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:44:02.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.382+0000 7f8ff1200640 1 -- 172.21.15.28:0/3819080988 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8fc806ac00 msgr2=0x7f8fc806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:02.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.382+0000 7f8ff1200640 1 --2- 172.21.15.28:0/3819080988 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8fc806ac00 0x7f8fc806d0c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0c2a10 tx=0x7f8fd4009190 comp rx=0 tx=0).stop 2024-09-13T16:44:02.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.382+0000 7f8ff1200640 1 -- 172.21.15.28:0/3819080988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec0c42f0 msgr2=0x7f8fec1520b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:02.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.382+0000 7f8ff1200640 1 --2- 172.21.15.28:0/3819080988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec0c42f0 0x7f8fec1520b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc02a2b0 tx=0x7f8fdc0046c0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.382+0000 7f8ff1200640 1 -- 172.21.15.28:0/3819080988 shutdown_connections 2024-09-13T16:44:02.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.382+0000 7f8ff1200640 1 --2- 172.21.15.28:0/3819080988 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8fc806ac00 0x7f8fc806d0c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.382+0000 7f8ff1200640 1 --2- 172.21.15.28:0/3819080988 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fec1ef410 0x7f8fec1ef8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.382+0000 7f8ff1200640 1 --2- 172.21.15.28:0/3819080988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fec14f790 0x7f8fec1e2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.382+0000 7f8ff1200640 1 --2- 172.21.15.28:0/3819080988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec0c42f0 0x7f8fec1520b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.382+0000 7f8ff1200640 1 -- 172.21.15.28:0/3819080988 >> 172.21.15.28:0/3819080988 conn(0x7f8fec0ca0e0 msgr2=0x7f8fec0c4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:02.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.382+0000 7f8ff1200640 1 -- 172.21.15.28:0/3819080988 shutdown_connections 2024-09-13T16:44:02.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.382+0000 7f8ff1200640 1 -- 172.21.15.28:0/3819080988 wait complete. 2024-09-13T16:44:02.401 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T16:44:02.401 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:44:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.554+0000 7f3556800640 1 Processor -- start 2024-09-13T16:44:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.554+0000 7f3556800640 1 -- start start 2024-09-13T16:44:02.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.555+0000 7f3556800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3550157480 0x7f355015cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:02.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.555+0000 7f3556800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35500c6a00 0x7f35500c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:02.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.555+0000 7f3556800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3550161400 con 0x7f35501570a0 2024-09-13T16:44:02.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.555+0000 7f3556800640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35501605e0 con 0x7f3550157480 2024-09-13T16:44:02.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.555+0000 7f3556800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3550160880 con 0x7f35500c6a00 2024-09-13T16:44:02.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.556+0000 7f354ea00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3550157480 0x7f355015cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:02.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.556+0000 7f354f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f35501570a0 0x7f35500c6ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59484/0 (socket says 172.21.15.28:59484) 2024-09-13T16:44:02.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.556+0000 7f354f400640 1 -- 172.21.15.28:0/739845400 learned_addr learned my addr 172.21.15.28:0/739845400 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:02.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.556+0000 7f354fe00640 1 --2- 172.21.15.28:0/739845400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35500c6a00 0x7f35500c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:02.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.556+0000 7f354ea00640 1 -- 172.21.15.28:0/739845400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35500c6a00 msgr2=0x7f35500c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:02.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.556+0000 7f354ea00640 1 --2- 172.21.15.28:0/739845400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35500c6a00 0x7f35500c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.556+0000 7f354ea00640 1 -- 172.21.15.28:0/739845400 >> v1:172.21.15.28:6789/0 conn(0x7f35501570a0 legacy=0x7f35500c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:02.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.556+0000 7f354ea00640 1 -- 172.21.15.28:0/739845400 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3550160b60 con 0x7f3550157480 2024-09-13T16:44:02.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.557+0000 7f354ea00640 1 --2- 172.21.15.28:0/739845400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3550157480 0x7f355015cf60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f353c0097a0 tx=0x7f353c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1029535151951c4a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:02.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.558+0000 7f354e000640 1 -- 172.21.15.28:0/739845400 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c032440 con 0x7f3550157480 2024-09-13T16:44:02.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.558+0000 7f354e000640 1 -- 172.21.15.28:0/739845400 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353c0325a0 con 0x7f3550157480 2024-09-13T16:44:02.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.558+0000 7f354e000640 1 -- 172.21.15.28:0/739845400 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c032930 con 0x7f3550157480 2024-09-13T16:44:02.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.558+0000 7f3556800640 1 -- 172.21.15.28:0/739845400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3550157480 msgr2=0x7f355015cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:02.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.558+0000 7f3556800640 1 --2- 172.21.15.28:0/739845400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3550157480 0x7f355015cf60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f353c0097a0 tx=0x7f353c02a180 comp rx=0 tx=0).stop 2024-09-13T16:44:02.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.558+0000 7f3556800640 1 -- 172.21.15.28:0/739845400 shutdown_connections 2024-09-13T16:44:02.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.558+0000 7f3556800640 1 --2- 172.21.15.28:0/739845400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35500c6a00 0x7f35500c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.558+0000 7f3556800640 1 --2- 172.21.15.28:0/739845400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3550157480 0x7f355015cf60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.558+0000 7f3556800640 1 -- 172.21.15.28:0/739845400 >> 172.21.15.28:0/739845400 conn(0x7f3550066040 msgr2=0x7f3550069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:02.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.558+0000 7f3556800640 1 -- 172.21.15.28:0/739845400 shutdown_connections 2024-09-13T16:44:02.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.558+0000 7f3556800640 1 -- 172.21.15.28:0/739845400 wait complete. 2024-09-13T16:44:02.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.560+0000 7f3556800640 1 Processor -- start 2024-09-13T16:44:02.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.560+0000 7f3556800640 1 -- start start 2024-09-13T16:44:02.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.561+0000 7f3556800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35500c6a00 0x7f35500be730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:02.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.561+0000 7f3556800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35501570a0 0x7f35501f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:02.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.561+0000 7f3556800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3550157480 0x7f35501519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:02.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.561+0000 7f3556800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3550151510 con 0x7f3550157480 2024-09-13T16:44:02.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.561+0000 7f3556800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3550150380 con 0x7f35500c6a00 2024-09-13T16:44:02.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.561+0000 7f3556800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3550150620 con 0x7f35501570a0 2024-09-13T16:44:02.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.561+0000 7f354fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3550157480 0x7f35501519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:02.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.561+0000 7f354fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3550157480 0x7f35501519a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41414/0 (socket says 172.21.15.28:41414) 2024-09-13T16:44:02.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.561+0000 7f354fe00640 1 -- 172.21.15.28:0/1446325681 learned_addr learned my addr 172.21.15.28:0/1446325681 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:02.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.561+0000 7f354f400640 1 --2- 172.21.15.28:0/1446325681 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35500c6a00 0x7f35500be730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:02.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.561+0000 7f354ea00640 1 --2- 172.21.15.28:0/1446325681 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35501570a0 0x7f35501f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:02.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.562+0000 7f354fe00640 1 -- 172.21.15.28:0/1446325681 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35501570a0 msgr2=0x7f35501f2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:02.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.562+0000 7f354fe00640 1 --2- 172.21.15.28:0/1446325681 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35501570a0 0x7f35501f2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.562+0000 7f354fe00640 1 -- 172.21.15.28:0/1446325681 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35500c6a00 msgr2=0x7f35500be730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:02.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.562+0000 7f354fe00640 1 --2- 172.21.15.28:0/1446325681 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35500c6a00 0x7f35500be730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.562+0000 7f354fe00640 1 -- 172.21.15.28:0/1446325681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353c009400 con 0x7f3550157480 2024-09-13T16:44:02.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.562+0000 7f354fe00640 1 --2- 172.21.15.28:0/1446325681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3550157480 0x7f35501519a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f354800efc0 tx=0x7f354800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:02.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.562+0000 7f3547400640 1 -- 172.21.15.28:0/1446325681 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3548009ba0 con 0x7f3550157480 2024-09-13T16:44:02.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.562+0000 7f3547400640 1 -- 172.21.15.28:0/1446325681 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354800f040 con 0x7f3550157480 2024-09-13T16:44:02.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.562+0000 7f3556800640 1 -- 172.21.15.28:0/1446325681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3550150900 con 0x7f3550157480 2024-09-13T16:44:02.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.562+0000 7f3547400640 1 -- 172.21.15.28:0/1446325681 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35480146c0 con 0x7f3550157480 2024-09-13T16:44:02.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.563+0000 7f3556800640 1 -- 172.21.15.28:0/1446325681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3550150ec0 con 0x7f3550157480 2024-09-13T16:44:02.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.564+0000 7f3556800640 1 -- 172.21.15.28:0/1446325681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3550151ee0 con 0x7f3550157480 2024-09-13T16:44:02.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.565+0000 7f3547400640 1 -- 172.21.15.28:0/1446325681 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3548014820 con 0x7f3550157480 2024-09-13T16:44:02.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.566+0000 7f3547400640 1 --2- 172.21.15.28:0/1446325681 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f353006ac30 0x7f353006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:02.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.566+0000 7f3547400640 1 -- 172.21.15.28:0/1446325681 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3548089df0 con 0x7f3550157480 2024-09-13T16:44:02.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.566+0000 7f354f400640 1 --2- 172.21.15.28:0/1446325681 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f353006ac30 0x7f353006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:02.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.567+0000 7f354f400640 1 --2- 172.21.15.28:0/1446325681 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f353006ac30 0x7f353006d0f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3540005880 tx=0x7f3540009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:02.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.570+0000 7f3547400640 1 -- 172.21.15.28:0/1446325681 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3548054f50 con 0x7f3550157480 2024-09-13T16:44:02.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.750+0000 7f3556800640 1 -- 172.21.15.28:0/1446325681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35501534b0 con 0x7f3550157480 2024-09-13T16:44:02.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.752+0000 7f3547400640 1 -- 172.21.15.28:0/1446325681 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16481 (secure 0 0 0) 0x7f3548018020 con 0x7f3550157480 2024-09-13T16:44:02.753 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:02.753 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:43:55.971358+0000","last_up_change":"2024-09-13T16:43:49.476313+0000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:43:49.643606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:43:52.828510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:43:53.945874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6801","nonce":2964323435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6803","nonce":2964323435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6807","nonce":2964323435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6805","nonce":2964323435}]},"public_addr":"172.21.15.28:6801/2964323435","cluster_addr":"172.21.15.28:6803/2964323435","heartbeat_back_addr":"172.21.15.28:6807/2964323435","heartbeat_front_addr":"172.21.15.28:6805/2964323435","state":["exists","up"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6809","nonce":23768570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6811","nonce":23768570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6815","nonce":23768570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6813","nonce":23768570}]},"public_addr":"172.21.15.28:6809/23768570","cluster_addr":"172.21.15.28:6811/23768570","heartbeat_back_addr":"172.21.15.28:6815/23768570","heartbeat_front_addr":"172.21.15.28:6813/23768570","state":["exists","up"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6817","nonce":886887303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6819","nonce":886887303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6823","nonce":886887303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6821","nonce":886887303}]},"public_addr":"172.21.15.28:6817/886887303","cluster_addr":"172.21.15.28:6819/886887303","heartbeat_back_addr":"172.21.15.28:6823/886887303","heartbeat_front_addr":"172.21.15.28:6821/886887303","state":["exists","up"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6825","nonce":2616036674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6827","nonce":2616036674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6831","nonce":2616036674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6829","nonce":2616036674}]},"public_addr":"172.21.15.28:6825/2616036674","cluster_addr":"172.21.15.28:6827/2616036674","heartbeat_back_addr":"172.21.15.28:6831/2616036674","heartbeat_front_addr":"172.21.15.28:6829/2616036674","state":["exists","up"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6801","nonce":2817102450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6803","nonce":2817102450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6807","nonce":2817102450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6805","nonce":2817102450}]},"public_addr":"172.21.15.191:6801/2817102450","cluster_addr":"172.21.15.191:6803/2817102450","heartbeat_back_addr":"172.21.15.191:6807/2817102450","heartbeat_front_addr":"172.21.15.191:6805/2817102450","state":["exists","up"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6817","nonce":2503027971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6819","nonce":2503027971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6823","nonce":2503027971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6821","nonce":2503027971}]},"public_addr":"172.21.15.191:6817/2503027971","cluster_addr":"172.21.15.191:6819/2503027971","heartbeat_back_addr":"172.21.15.191:6823/2503027971","heartbeat_front_addr":"172.21.15.191:6821/2503027971","state":["exists","up"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6809","nonce":3006595088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6811","nonce":3006595088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6815","nonce":3006595088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6813","nonce":3006595088}]},"public_addr":"172.21.15.191:6809/3006595088","cluster_addr":"172.21.15.191:6811/3006595088","heartbeat_back_addr":"172.21.15.191:6815/3006595088","heartbeat_front_addr":"172.21.15.191:6813/3006595088","state":["exists","up"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6825","nonce":1477327720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6827","nonce":1477327720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6831","nonce":1477327720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6829","nonce":1477327720}]},"public_addr":"172.21.15.191:6825/1477327720","cluster_addr":"172.21.15.191:6827/1477327720","heartbeat_back_addr":"172.21.15.191:6831/1477327720","heartbeat_front_addr":"172.21.15.191:6829/1477327720","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.917250+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.954351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.017329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.046221+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.707467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.726947+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.782321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.764850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:44:02.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.754+0000 7f3556800640 1 -- 172.21.15.28:0/1446325681 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f353006ac30 msgr2=0x7f353006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:02.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.754+0000 7f3556800640 1 --2- 172.21.15.28:0/1446325681 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f353006ac30 0x7f353006d0f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3540005880 tx=0x7f3540009040 comp rx=0 tx=0).stop 2024-09-13T16:44:02.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.754+0000 7f3556800640 1 -- 172.21.15.28:0/1446325681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3550157480 msgr2=0x7f35501519a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:02.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.754+0000 7f3556800640 1 --2- 172.21.15.28:0/1446325681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3550157480 0x7f35501519a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f354800efc0 tx=0x7f354800c490 comp rx=0 tx=0).stop 2024-09-13T16:44:02.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.754+0000 7f3556800640 1 -- 172.21.15.28:0/1446325681 shutdown_connections 2024-09-13T16:44:02.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.754+0000 7f3556800640 1 --2- 172.21.15.28:0/1446325681 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f353006ac30 0x7f353006d0f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.754+0000 7f3556800640 1 --2- 172.21.15.28:0/1446325681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3550157480 0x7f35501519a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.754+0000 7f3556800640 1 --2- 172.21.15.28:0/1446325681 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f35501570a0 0x7f35501f2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.754+0000 7f3556800640 1 --2- 172.21.15.28:0/1446325681 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35500c6a00 0x7f35500be730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:02.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.754+0000 7f3556800640 1 -- 172.21.15.28:0/1446325681 >> 172.21.15.28:0/1446325681 conn(0x7f3550066040 msgr2=0x7f35500c8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:02.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.754+0000 7f3556800640 1 -- 172.21.15.28:0/1446325681 shutdown_connections 2024-09-13T16:44:02.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:02.754+0000 7f3556800640 1 -- 172.21.15.28:0/1446325681 wait complete. 2024-09-13T16:44:02.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T16:44:02.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T16:44:02.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T16:44:02.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-13T16:44:02.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-13T16:44:02.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-13T16:44:02.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-13T16:44:02.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-13T16:44:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.047+0000 7fb23cc00640 1 Processor -- start 2024-09-13T16:44:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.047+0000 7fb23cc00640 1 -- start start 2024-09-13T16:44:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.048+0000 7f89c7600640 1 Processor -- start 2024-09-13T16:44:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.048+0000 7f89c7600640 1 -- start start 2024-09-13T16:44:03.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.050+0000 7f89c7600640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f89b80d8830 0x7f89b8049910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.050+0000 7f89c7600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b8050810 con 0x7f89b80dee00 2024-09-13T16:44:03.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.050+0000 7f89c7600640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b80e1c20 con 0x7f89b8050290 2024-09-13T16:44:03.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.050+0000 7f89c7600640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89b80e28a0 con 0x7f89b80d8830 2024-09-13T16:44:03.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.050+0000 7fb23cc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb2380ce5e0 0x7fb2381595a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.051+0000 7f89c5600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f89b80dee00 0x7f89b80e8f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59494/0 (socket says 172.21.15.28:59494) 2024-09-13T16:44:03.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.051+0000 7f89c5600640 1 -- 172.21.15.28:0/769194324 learned_addr learned my addr 172.21.15.28:0/769194324 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:03.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.051+0000 7f89c4c00640 1 --2- 172.21.15.28:0/769194324 >> v2:172.21.15.191:3300/0 conn(0x7f89b80d8830 0x7f89b8049910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.052+0000 7f89bf400640 1 -- 172.21.15.28:0/769194324 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288937184 0 0) 0x7f89b8050810 con 0x7f89b80dee00 2024-09-13T16:44:03.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.052+0000 7f89bf400640 1 -- 172.21.15.28:0/769194324 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a8003520 con 0x7f89b80dee00 2024-09-13T16:44:03.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.051+0000 7fb23cc00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2380c94d0 0x7fb2380c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.051+0000 7fb23cc00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb238159f90 con 0x7fb2380ce9c0 2024-09-13T16:44:03.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.051+0000 7fb23cc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb238159ae0 con 0x7fb2380ce5e0 2024-09-13T16:44:03.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.051+0000 7fb23cc00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2380c6900 con 0x7fb2380c94d0 2024-09-13T16:44:03.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.051+0000 7fb236a00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fb2380ce9c0 0x7fb23815b590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:51458/0 (socket says 172.21.15.28:51458) 2024-09-13T16:44:03.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.052+0000 7fb236a00640 1 -- 172.21.15.28:0/171246515 learned_addr learned my addr 172.21.15.28:0/171246515 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:03.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.052+0000 7fb236000640 1 -- 172.21.15.28:0/171246515 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425192073 0 0) 0x7fb238159f90 con 0x7fb2380ce9c0 2024-09-13T16:44:03.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.052+0000 7fb236000640 1 -- 172.21.15.28:0/171246515 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb21c003520 con 0x7fb2380ce9c0 2024-09-13T16:44:03.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.052+0000 7f89bf400640 1 -- 172.21.15.28:0/769194324 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271501561 0 0) 0x7f89b80e1c20 con 0x7f89b8050290 2024-09-13T16:44:03.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.052+0000 7f89bf400640 1 -- 172.21.15.28:0/769194324 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89b8050810 con 0x7f89b8050290 2024-09-13T16:44:03.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.053+0000 7f89c4c00640 1 -- 172.21.15.28:0/769194324 >> v1:172.21.15.191:6789/0 conn(0x7f89b8050290 legacy=0x7f89b80de6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.053+0000 7f89c4c00640 1 -- 172.21.15.28:0/769194324 >> v1:172.21.15.28:6789/0 conn(0x7f89b80dee00 legacy=0x7f89b80e8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.053+0000 7f89c4c00640 1 -- 172.21.15.28:0/769194324 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b80e1f00 con 0x7f89b80d8830 2024-09-13T16:44:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.051+0000 7fb237400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb2380ce5e0 0x7fb2381595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.053+0000 7fb236000640 1 -- 172.21.15.28:0/171246515 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3757160560 0 0) 0x7fb21c003520 con 0x7fb2380ce9c0 2024-09-13T16:44:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.053+0000 7fb236000640 1 -- 172.21.15.28:0/171246515 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2380c94d0 msgr2=0x7fb2380c6d90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-13T16:44:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.053+0000 7fb236000640 1 --2- 172.21.15.28:0/171246515 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2380c94d0 0x7fb2380c6d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.053+0000 7fb236000640 1 -- 172.21.15.28:0/171246515 >> v2:172.21.15.28:3300/0 conn(0x7fb2380ce5e0 msgr2=0x7fb2381595a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.053+0000 7fb236000640 1 --2- 172.21.15.28:0/171246515 >> v2:172.21.15.28:3300/0 conn(0x7fb2380ce5e0 0x7fb2381595a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.053+0000 7fb236000640 1 -- 172.21.15.28:0/171246515 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2380c5800 con 0x7fb2380ce9c0 2024-09-13T16:44:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.053+0000 7fb236000640 1 -- 172.21.15.28:0/171246515 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1654382301 0 0) 0x7fb228001650 con 0x7fb2380ce9c0 2024-09-13T16:44:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.053+0000 7fb236000640 1 -- 172.21.15.28:0/171246515 >> v1:172.21.15.191:6789/0 conn(0x7fb2380ce9c0 legacy=0x7fb23815b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.053+0000 7fb236000640 1 --2- 172.21.15.28:0/171246515 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb21c003990 0x7fb21c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.053+0000 7fb236000640 1 --2- 172.21.15.28:0/171246515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb21c0047c0 0x7fb21c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.053+0000 7fb236000640 1 --2- 172.21.15.28:0/171246515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb21c0052e0 0x7fb21c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.053+0000 7fb236000640 1 -- 172.21.15.28:0/171246515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb238159ae0 con 0x7fb21c0052e0 2024-09-13T16:44:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.053+0000 7fb236000640 1 -- 172.21.15.28:0/171246515 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2380c6900 con 0x7fb21c0047c0 2024-09-13T16:44:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.053+0000 7fb236000640 1 -- 172.21.15.28:0/171246515 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c00c260 con 0x7fb21c003990 2024-09-13T16:44:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.054+0000 7fb237400640 1 --2- 172.21.15.28:0/171246515 >> v2:172.21.15.28:3300/0 conn(0x7fb2380ce5e0 0x7fb2381595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:44:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.054+0000 7fb236a00640 1 --2- 172.21.15.28:0/171246515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb21c0047c0 0x7fb21c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.054+0000 7fb237400640 1 --2- 172.21.15.28:0/171246515 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb21c003990 0x7fb21c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.054+0000 7f7890a00640 1 Processor -- start 2024-09-13T16:44:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.055+0000 7f89c4c00640 1 --2- 172.21.15.28:0/769194324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89b80d8830 0x7f89b8049910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f89b40097a0 tx=0x7f89b402a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5c87b3e830752cf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.055+0000 7fb237400640 1 -- 172.21.15.28:0/171246515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb21c0047c0 msgr2=0x7fb21c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.055+0000 7fb237400640 1 --2- 172.21.15.28:0/171246515 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb21c0047c0 0x7fb21c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.055+0000 7fb237400640 1 -- 172.21.15.28:0/171246515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb21c0052e0 msgr2=0x7fb21c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.055+0000 7fb237400640 1 --2- 172.21.15.28:0/171246515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb21c0052e0 0x7fb21c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.055+0000 7fb237400640 1 -- 172.21.15.28:0/171246515 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21c003520 con 0x7fb21c003990 2024-09-13T16:44:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.055+0000 7f7890a00640 1 -- start start 2024-09-13T16:44:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.056+0000 7f89bf400640 1 -- 172.21.15.28:0/769194324 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b4032440 con 0x7f89b80d8830 2024-09-13T16:44:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.056+0000 7f89bf400640 1 -- 172.21.15.28:0/769194324 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b40325a0 con 0x7f89b80d8830 2024-09-13T16:44:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.056+0000 7f89bf400640 1 -- 172.21.15.28:0/769194324 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b4032930 con 0x7f89b80d8830 2024-09-13T16:44:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.056+0000 7fb237400640 1 --2- 172.21.15.28:0/171246515 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb21c003990 0x7fb21c003dd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb23000c490 tx=0x7fb21c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a16946460766d4fe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.057+0000 7f7890a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78840dcd60 0x7f7884045d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.057+0000 7f7890a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f78840e6610 0x7f788404aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.057+0000 7f7890a00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7884044d10 con 0x7f788404a650 2024-09-13T16:44:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.057+0000 7f7890a00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788404af90 con 0x7f78840e6610 2024-09-13T16:44:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.057+0000 7f7890a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78840dd140 con 0x7f78840dcd60 2024-09-13T16:44:03.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.057+0000 7f89c7600640 1 -- 172.21.15.28:0/769194324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89b80d8830 msgr2=0x7f89b8049910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.057+0000 7f89c7600640 1 --2- 172.21.15.28:0/769194324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89b80d8830 0x7f89b8049910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f89b40097a0 tx=0x7f89b402a1e0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.057+0000 7f788a800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78840dcd60 0x7f7884045d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.057+0000 7f788b200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f78840e6610 0x7f788404aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.057+0000 7f788a800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78840dcd60 0x7f7884045d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37340/0 (socket says 172.21.15.28:37340) 2024-09-13T16:44:03.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.057+0000 7f788a800640 1 -- 172.21.15.28:0/3676507341 learned_addr learned my addr 172.21.15.28:0/3676507341 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.058+0000 7fb236000640 1 -- 172.21.15.28:0/171246515 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23002bbb0 con 0x7fb21c003990 2024-09-13T16:44:03.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.058+0000 7f89c7600640 1 -- 172.21.15.28:0/769194324 shutdown_connections 2024-09-13T16:44:03.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.058+0000 7f89c7600640 1 --2- 172.21.15.28:0/769194324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89b80d8830 0x7f89b8049910 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.058+0000 7f89c7600640 1 -- 172.21.15.28:0/769194324 >> 172.21.15.28:0/769194324 conn(0x7f89b8054410 msgr2=0x7f89b8054820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.058+0000 7fb236000640 1 -- 172.21.15.28:0/171246515 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23002bd10 con 0x7fb21c003990 2024-09-13T16:44:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.058+0000 7fb236000640 1 -- 172.21.15.28:0/171246515 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb230033680 con 0x7fb21c003990 2024-09-13T16:44:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.058+0000 7f89c7600640 1 -- 172.21.15.28:0/769194324 shutdown_connections 2024-09-13T16:44:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.058+0000 7f788a800640 1 -- 172.21.15.28:0/3676507341 >> v2:172.21.15.191:3300/0 conn(0x7f78840e6610 msgr2=0x7f788404aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.058+0000 7f788a800640 1 --2- 172.21.15.28:0/3676507341 >> v2:172.21.15.191:3300/0 conn(0x7f78840e6610 0x7f788404aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.058+0000 7f788a800640 1 -- 172.21.15.28:0/3676507341 >> v1:172.21.15.191:6789/0 conn(0x7f788404a650 legacy=0x7f78840e5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.058+0000 7f788a800640 1 -- 172.21.15.28:0/3676507341 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78840e6aa0 con 0x7f78840dcd60 2024-09-13T16:44:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.058+0000 7f788b200640 1 --2- 172.21.15.28:0/3676507341 >> v2:172.21.15.191:3300/0 conn(0x7f78840e6610 0x7f788404aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.058+0000 7f89c7600640 1 -- 172.21.15.28:0/769194324 wait complete. 2024-09-13T16:44:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.059+0000 7fb23cc00640 1 -- 172.21.15.28:0/171246515 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb21c003990 msgr2=0x7fb21c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.059+0000 7fb23cc00640 1 --2- 172.21.15.28:0/171246515 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb21c003990 0x7fb21c003dd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb23000c490 tx=0x7fb21c0057f0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.059+0000 7fb237e00640 1 -- 172.21.15.28:0/171246515 reap_dead start 2024-09-13T16:44:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.059+0000 7fb23cc00640 1 -- 172.21.15.28:0/171246515 shutdown_connections 2024-09-13T16:44:03.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.059+0000 7fb23cc00640 1 -- 172.21.15.28:0/171246515 >> 172.21.15.28:0/171246515 conn(0x7fb23806c880 msgr2=0x7fb2380715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.059+0000 7fb23cc00640 1 -- 172.21.15.28:0/171246515 shutdown_connections 2024-09-13T16:44:03.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.060+0000 7f788a800640 1 --2- 172.21.15.28:0/3676507341 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78840dcd60 0x7f7884045d20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f787c009870 tx=0x7f787c02a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18ed7c11bb236e6d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.059+0000 7fb23cc00640 1 -- 172.21.15.28:0/171246515 wait complete. 2024-09-13T16:44:03.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.060+0000 7f7889400640 1 -- 172.21.15.28:0/3676507341 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c032640 con 0x7f78840dcd60 2024-09-13T16:44:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.060+0000 7f7889400640 1 -- 172.21.15.28:0/3676507341 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787c0327a0 con 0x7f78840dcd60 2024-09-13T16:44:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.060+0000 7f7889400640 1 -- 172.21.15.28:0/3676507341 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c032b30 con 0x7f78840dcd60 2024-09-13T16:44:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.061+0000 7f89c7600640 1 Processor -- start 2024-09-13T16:44:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.061+0000 7f89c7600640 1 -- start start 2024-09-13T16:44:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.062+0000 7f7890a00640 1 -- 172.21.15.28:0/3676507341 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78840dcd60 msgr2=0x7f7884045d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.062+0000 7f7890a00640 1 --2- 172.21.15.28:0/3676507341 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78840dcd60 0x7f7884045d20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f787c009870 tx=0x7f787c02a300 comp rx=0 tx=0).stop 2024-09-13T16:44:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.062+0000 7f7890a00640 1 -- 172.21.15.28:0/3676507341 shutdown_connections 2024-09-13T16:44:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.062+0000 7f7890a00640 1 --2- 172.21.15.28:0/3676507341 >> v2:172.21.15.191:3300/0 conn(0x7f78840e6610 0x7f788404aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.062+0000 7f7890a00640 1 --2- 172.21.15.28:0/3676507341 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78840dcd60 0x7f7884045d20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.062+0000 7f7890a00640 1 -- 172.21.15.28:0/3676507341 >> 172.21.15.28:0/3676507341 conn(0x7f788404fc20 msgr2=0x7f7884050030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.062+0000 7fb23cc00640 1 Processor -- start 2024-09-13T16:44:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.062+0000 7fb23cc00640 1 -- start start 2024-09-13T16:44:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.062+0000 7f89c7600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89b8050290 0x7f89b80e0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.063+0000 7f89c7600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89b80d8830 0x7f89b80dc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.063+0000 7f89c7600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89b80dee00 0x7f89b80dc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.063+0000 7f89c7600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b80e1460 con 0x7f89b80dee00 2024-09-13T16:44:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.063+0000 7f89c7600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b80dfdb0 con 0x7f89b8050290 2024-09-13T16:44:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.063+0000 7f89c7600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b80dda80 con 0x7f89b80d8830 2024-09-13T16:44:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.062+0000 7f7890a00640 1 -- 172.21.15.28:0/3676507341 shutdown_connections 2024-09-13T16:44:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.062+0000 7f7890a00640 1 -- 172.21.15.28:0/3676507341 wait complete. 2024-09-13T16:44:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.064+0000 7fb23cc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb21c003990 0x7fb23814eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.064+0000 7fb23cc00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2380c1180 0x7fb2380c1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.065+0000 7f89c5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89b80dee00 0x7f89b80dc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.065+0000 7fb237400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb21c003990 0x7fb23814eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.065+0000 7fb236a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2380c1180 0x7fb2380c1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.065+0000 7fb236a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2380c1180 0x7fb2380c1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37356/0 (socket says 172.21.15.28:37356) 2024-09-13T16:44:03.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.065+0000 7fb236a00640 1 -- 172.21.15.28:0/828383813 learned_addr learned my addr 172.21.15.28:0/828383813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.065+0000 7f7890a00640 1 Processor -- start 2024-09-13T16:44:03.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.065+0000 7f7890a00640 1 -- start start 2024-09-13T16:44:03.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.065+0000 7fb23cc00640 1 --2- 172.21.15.28:0/828383813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2380c2470 0x7fb23814cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.065+0000 7fb23cc00640 1 -- 172.21.15.28:0/828383813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb238154a40 con 0x7fb2380c2470 2024-09-13T16:44:03.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.065+0000 7fb23cc00640 1 -- 172.21.15.28:0/828383813 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2381570b0 con 0x7fb21c003990 2024-09-13T16:44:03.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.066+0000 7fb23cc00640 1 -- 172.21.15.28:0/828383813 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2380c42b0 con 0x7fb2380c1180 2024-09-13T16:44:03.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.065+0000 7f89bfe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89b80d8830 0x7f89b80dc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.065+0000 7f89bfe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89b80d8830 0x7f89b80dc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37354/0 (socket says 172.21.15.28:37354) 2024-09-13T16:44:03.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.065+0000 7f89bfe00640 1 -- 172.21.15.28:0/1085338538 learned_addr learned my addr 172.21.15.28:0/1085338538 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.066+0000 7f89c4c00640 1 --2- 172.21.15.28:0/1085338538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89b8050290 0x7f89b80e0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.066+0000 7fb237e00640 1 --2- 172.21.15.28:0/828383813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2380c2470 0x7fb23814cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.066+0000 7f89bfe00640 1 -- 172.21.15.28:0/1085338538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89b8050290 msgr2=0x7f89b80e0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.066+0000 7f89bfe00640 1 --2- 172.21.15.28:0/1085338538 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89b8050290 0x7f89b80e0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7f7890a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f788404a650 0x7f7884046f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7f7890a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78840dcd60 0x7f78840e04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7f7890a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78840e6610 0x7f78840dbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7f7889e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78840dcd60 0x7f78840e04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7f7889e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78840dcd60 0x7f78840e04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37362/0 (socket says 172.21.15.28:37362) 2024-09-13T16:44:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7f788a800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f788404a650 0x7f7884046f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7f788a800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f788404a650 0x7f7884046f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60604/0 (socket says 172.21.15.28:60604) 2024-09-13T16:44:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7f7890a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78840e1c70 con 0x7f78840e6610 2024-09-13T16:44:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7f7890a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7884049e40 con 0x7f788404a650 2024-09-13T16:44:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7f7890a00640 1 -- 172.21.15.28:0/2990707434 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78840e1340 con 0x7f78840dcd60 2024-09-13T16:44:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7f7889e00640 1 -- 172.21.15.28:0/2990707434 learned_addr learned my addr 172.21.15.28:0/2990707434 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.066+0000 7f89bfe00640 1 -- 172.21.15.28:0/1085338538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89b80dee00 msgr2=0x7f89b80dc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.066+0000 7f89bfe00640 1 --2- 172.21.15.28:0/1085338538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89b80dee00 0x7f89b80dc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.066+0000 7f89bfe00640 1 -- 172.21.15.28:0/1085338538 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b4009400 con 0x7f89b80d8830 2024-09-13T16:44:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.066+0000 7f89c5600640 1 --2- 172.21.15.28:0/1085338538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89b80dee00 0x7f89b80dc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7f89bfe00640 1 --2- 172.21.15.28:0/1085338538 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89b80d8830 0x7f89b80dc250 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00c500 tx=0x7f89ac00c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.068+0000 7f788b200640 1 --2- 172.21.15.28:0/2990707434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78840e6610 0x7f78840dbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.068+0000 7f89bcc00640 1 -- 172.21.15.28:0/1085338538 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ac0035f0 con 0x7f89b80d8830 2024-09-13T16:44:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.068+0000 7f89bcc00640 1 -- 172.21.15.28:0/1085338538 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ac003750 con 0x7f89b80d8830 2024-09-13T16:44:03.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.068+0000 7f89bcc00640 1 -- 172.21.15.28:0/1085338538 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ac0035f0 con 0x7f89b80d8830 2024-09-13T16:44:03.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7fb237400640 1 -- 172.21.15.28:0/828383813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2380c1180 msgr2=0x7fb2380c1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7fb237400640 1 --2- 172.21.15.28:0/828383813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2380c1180 0x7fb2380c1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7fb237400640 1 -- 172.21.15.28:0/828383813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2380c2470 msgr2=0x7fb23814cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7fb237400640 1 --2- 172.21.15.28:0/828383813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2380c2470 0x7fb23814cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7fb237400640 1 -- 172.21.15.28:0/828383813 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb238154c50 con 0x7fb21c003990 2024-09-13T16:44:03.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.067+0000 7fb236a00640 1 --2- 172.21.15.28:0/828383813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2380c1180 0x7fb2380c1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:03.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.068+0000 7fb237400640 1 --2- 172.21.15.28:0/828383813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb21c003990 0x7fb23814eb80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb230032000 tx=0x7fb230004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.068+0000 7fb22f400640 1 -- 172.21.15.28:0/828383813 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23003d610 con 0x7fb21c003990 2024-09-13T16:44:03.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.068+0000 7fb23cc00640 1 -- 172.21.15.28:0/828383813 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb238154f30 con 0x7fb21c003990 2024-09-13T16:44:03.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.069+0000 7fb23cc00640 1 -- 172.21.15.28:0/828383813 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2381554f0 con 0x7fb21c003990 2024-09-13T16:44:03.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.069+0000 7f788b200640 1 -- 172.21.15.28:0/2990707434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78840dcd60 msgr2=0x7f78840e04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.069+0000 7f89c7600640 1 -- 172.21.15.28:0/1085338538 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b80ddd20 con 0x7f89b80d8830 2024-09-13T16:44:03.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.069+0000 7f89c7600640 1 -- 172.21.15.28:0/1085338538 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b80dfa40 con 0x7f89b80d8830 2024-09-13T16:44:03.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.069+0000 7f788b200640 1 --2- 172.21.15.28:0/2990707434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78840dcd60 0x7f78840e04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.069+0000 7f788b200640 1 -- 172.21.15.28:0/2990707434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f788404a650 msgr2=0x7f7884046f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.069+0000 7f788b200640 1 --2- 172.21.15.28:0/2990707434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f788404a650 0x7f7884046f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.069+0000 7f788b200640 1 -- 172.21.15.28:0/2990707434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c075340 con 0x7f78840e6610 2024-09-13T16:44:03.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.069+0000 7f788a800640 1 --2- 172.21.15.28:0/2990707434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f788404a650 0x7f7884046f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:03.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.069+0000 7f7889e00640 1 --2- 172.21.15.28:0/2990707434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78840dcd60 0x7f78840e04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:03.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.069+0000 7fb22f400640 1 -- 172.21.15.28:0/828383813 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23003d770 con 0x7fb21c003990 2024-09-13T16:44:03.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.069+0000 7fb22f400640 1 -- 172.21.15.28:0/828383813 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23003d610 con 0x7fb21c003990 2024-09-13T16:44:03.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.070+0000 7fa413200640 1 Processor -- start 2024-09-13T16:44:03.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.070+0000 7fb23cc00640 1 -- 172.21.15.28:0/828383813 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb210000fc0 con 0x7fb21c003990 2024-09-13T16:44:03.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.071+0000 7fa413200640 1 -- start start 2024-09-13T16:44:03.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.071+0000 7f89c7600640 1 -- 172.21.15.28:0/1085338538 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f89b800f6e0 con 0x7f89b80d8830 2024-09-13T16:44:03.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.070+0000 7f788b200640 1 --2- 172.21.15.28:0/2990707434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78840e6610 0x7f78840dbf10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f788c063830 tx=0x7f788c067a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.071+0000 7f7882a00640 1 -- 172.21.15.28:0/2990707434 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c06f810 con 0x7f78840e6610 2024-09-13T16:44:03.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.071+0000 7fe266a00640 1 Processor -- start 2024-09-13T16:44:03.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.071+0000 7f7890a00640 1 -- 172.21.15.28:0/2990707434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f787c0094d0 con 0x7f78840e6610 2024-09-13T16:44:03.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.071+0000 7f7890a00640 1 -- 172.21.15.28:0/2990707434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78840e1900 con 0x7f78840e6610 2024-09-13T16:44:03.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.071+0000 7fe266a00640 1 -- start start 2024-09-13T16:44:03.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.073+0000 7fa413200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40c15c150 0x7fa40c15c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.073+0000 7fa413200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa40c0c3bc0 con 0x7fa40c152f20 2024-09-13T16:44:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.073+0000 7fa413200640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa40c15d0e0 con 0x7fa40c0ca5d0 2024-09-13T16:44:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.073+0000 7fa413200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa40c15ce00 con 0x7fa40c15c150 2024-09-13T16:44:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.072+0000 7f7882a00640 1 -- 172.21.15.28:0/2990707434 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788c06ae90 con 0x7f78840e6610 2024-09-13T16:44:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.072+0000 7f7882a00640 1 -- 172.21.15.28:0/2990707434 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c06f810 con 0x7f78840e6610 2024-09-13T16:44:03.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.073+0000 7fe266a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2600c25c0 0x7fe260071410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.073+0000 7fe266a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe260071950 0x7fe2600c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.073+0000 7fe266a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe2600c74a0 0x7fe26006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.073+0000 7fe266a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2600c1d00 con 0x7fe2600c25c0 2024-09-13T16:44:03.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.073+0000 7fe266a00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2600c5e70 con 0x7fe2600c74a0 2024-09-13T16:44:03.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.073+0000 7fe266a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2601659e0 con 0x7fe260071950 2024-09-13T16:44:03.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.073+0000 7fa411200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40c15c150 0x7fa40c15c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.073+0000 7fa411200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40c15c150 0x7fa40c15c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37364/0 (socket says 172.21.15.28:37364) 2024-09-13T16:44:03.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.074+0000 7fa411200640 1 -- 172.21.15.28:0/4242220132 learned_addr learned my addr 172.21.15.28:0/4242220132 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.074+0000 7fe25f400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe260071950 0x7fe2600c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.074+0000 7fe25f400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe260071950 0x7fe2600c6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37374/0 (socket says 172.21.15.28:37374) 2024-09-13T16:44:03.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.074+0000 7fe25f400640 1 -- 172.21.15.28:0/538826190 learned_addr learned my addr 172.21.15.28:0/538826190 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.074+0000 7fe264a00640 1 --2- 172.21.15.28:0/538826190 >> v2:172.21.15.191:3300/0 conn(0x7fe2600c74a0 0x7fe26006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.074+0000 7fa411200640 1 -- 172.21.15.28:0/4242220132 >> v1:172.21.15.191:6789/0 conn(0x7fa40c0ca5d0 legacy=0x7fa40c0cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.074+0000 7fa411200640 1 -- 172.21.15.28:0/4242220132 >> v1:172.21.15.28:6789/0 conn(0x7fa40c152f20 legacy=0x7fa40c0c34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.074+0000 7f7890a00640 1 -- 172.21.15.28:0/2990707434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7864000fc0 con 0x7f78840e6610 2024-09-13T16:44:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.075+0000 7fb22f400640 1 -- 172.21.15.28:0/828383813 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb23003a050 con 0x7fb21c003990 2024-09-13T16:44:03.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.075+0000 7fe25fe00640 1 --2- 172.21.15.28:0/538826190 >> v2:172.21.15.28:3300/0 conn(0x7fe2600c25c0 0x7fe260071410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.073+0000 7f599f600640 1 Processor -- start 2024-09-13T16:44:03.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.073+0000 7f599f600640 1 -- start start 2024-09-13T16:44:03.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.075+0000 7f599f600640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5998157110 0x7f59980c5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.075+0000 7f599f600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f599806df90 0x7f59980c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.075+0000 7f599f600640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599815dee0 con 0x7f59981574f0 2024-09-13T16:44:03.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7f599f600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5998071c60 con 0x7f599806df90 2024-09-13T16:44:03.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7f599f600640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59980c1ff0 con 0x7f5998157110 2024-09-13T16:44:03.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7f599cc00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5998157110 0x7f59980c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7f599cc00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5998157110 0x7f59980c5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60618/0 (socket says 172.21.15.28:60618) 2024-09-13T16:44:03.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7f599cc00640 1 -- 172.21.15.28:0/3203222142 learned_addr learned my addr 172.21.15.28:0/3203222142 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7fe25f400640 1 -- 172.21.15.28:0/538826190 >> v2:172.21.15.191:3300/0 conn(0x7fe2600c74a0 msgr2=0x7fe26006d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7fe25f400640 1 --2- 172.21.15.28:0/538826190 >> v2:172.21.15.191:3300/0 conn(0x7fe2600c74a0 0x7fe26006d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7fe25f400640 1 -- 172.21.15.28:0/538826190 >> v2:172.21.15.28:3300/0 conn(0x7fe2600c25c0 msgr2=0x7fe260071410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7fb22f400640 1 --2- 172.21.15.28:0/828383813 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb2180734a0 0x7fb218075960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.075+0000 7fa411200640 1 -- 172.21.15.28:0/4242220132 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40c06ad50 con 0x7fa40c15c150 2024-09-13T16:44:03.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7f599d600640 1 --2- 172.21.15.28:0/3203222142 >> v2:172.21.15.28:3300/0 conn(0x7f599806df90 0x7f59980c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7fe25f400640 1 --2- 172.21.15.28:0/538826190 >> v2:172.21.15.28:3300/0 conn(0x7fe2600c25c0 0x7fe260071410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7fe25f400640 1 -- 172.21.15.28:0/538826190 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe260165f30 con 0x7fe260071950 2024-09-13T16:44:03.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7fe25fe00640 1 --2- 172.21.15.28:0/538826190 >> v2:172.21.15.28:3300/0 conn(0x7fe2600c25c0 0x7fe260071410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:03.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7fe264a00640 1 --2- 172.21.15.28:0/538826190 >> v2:172.21.15.191:3300/0 conn(0x7fe2600c74a0 0x7fe26006d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:44:03.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7f7882a00640 1 -- 172.21.15.28:0/2990707434 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f788c06f990 con 0x7f78840e6610 2024-09-13T16:44:03.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.077+0000 7fa411200640 1 --2- 172.21.15.28:0/4242220132 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40c15c150 0x7fa40c15c580 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa404009e80 tx=0x7fa40402a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=161816a2368c0aca server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7f09e2a00640 1 Processor -- start 2024-09-13T16:44:03.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7f09e2a00640 1 -- start start 2024-09-13T16:44:03.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.076+0000 7fb22f400640 1 -- 172.21.15.28:0/828383813 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2300aaa10 con 0x7fb21c003990 2024-09-13T16:44:03.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.077+0000 7fb22f400640 1 --2- 172.21.15.28:0/828383813 >> [v2:172.21.15.28:6808/23768570,v1:172.21.15.28:6809/23768570] conn(0x7fb21807b6c0 0x7fb21807dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.077+0000 7fb22f400640 1 -- 172.21.15.28:0/828383813 --> [v2:172.21.15.28:6808/23768570,v1:172.21.15.28:6809/23768570] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb21807e190 con 0x7fb21807b6c0 2024-09-13T16:44:03.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.077+0000 7fb22f400640 1 -- 172.21.15.28:0/828383813 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb23006a020 con 0x7fb21c003990 2024-09-13T16:44:03.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.077+0000 7fb236a00640 1 --2- 172.21.15.28:0/828383813 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb2180734a0 0x7fb218075960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.077+0000 7f89bcc00640 1 -- 172.21.15.28:0/1085338538 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f89ac0038c0 con 0x7f89b80d8830 2024-09-13T16:44:03.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.077+0000 7f599d600640 1 -- 172.21.15.28:0/3203222142 >> v2:172.21.15.191:3300/0 conn(0x7f5998157110 msgr2=0x7f59980c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.077+0000 7f599d600640 1 --2- 172.21.15.28:0/3203222142 >> v2:172.21.15.191:3300/0 conn(0x7f5998157110 0x7f59980c5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.077+0000 7f599d600640 1 -- 172.21.15.28:0/3203222142 >> v1:172.21.15.191:6789/0 conn(0x7f59981574f0 legacy=0x7f599806d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.077+0000 7f599d600640 1 -- 172.21.15.28:0/3203222142 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599806d3f0 con 0x7f599806df90 2024-09-13T16:44:03.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.078+0000 7fa40aa00640 1 -- 172.21.15.28:0/4242220132 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa404031600 con 0x7fa40c15c150 2024-09-13T16:44:03.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.077+0000 7f4e63200640 1 Processor -- start 2024-09-13T16:44:03.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.077+0000 7f4e63200640 1 -- start start 2024-09-13T16:44:03.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.078+0000 7fa40aa00640 1 -- 172.21.15.28:0/4242220132 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40402d050 con 0x7fa40c15c150 2024-09-13T16:44:03.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.078+0000 7fa40aa00640 1 -- 172.21.15.28:0/4242220132 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa404031980 con 0x7fa40c15c150 2024-09-13T16:44:03.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.078+0000 7fb236a00640 1 --2- 172.21.15.28:0/828383813 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb2180734a0 0x7fb218075960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb228007fd0 tx=0x7fb228001d80 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.078+0000 7fb237e00640 1 --2- 172.21.15.28:0/828383813 >> [v2:172.21.15.28:6808/23768570,v1:172.21.15.28:6809/23768570] conn(0x7fb21807b6c0 0x7fb21807dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.078+0000 7f7882a00640 1 --2- 172.21.15.28:0/2990707434 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f786c06ac60 0x7f786c06d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.078+0000 7f09e2a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc0c75b0 0x7f09dc1580b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.078+0000 7fe25f400640 1 --2- 172.21.15.28:0/538826190 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe260071950 0x7fe2600c6f60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe26008ac00 tx=0x7fe24c02a1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45e26d509165190d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.078+0000 7f09e2a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09dc06d780 con 0x7f09dc0c65f0 2024-09-13T16:44:03.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7f09e2a00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09dc158810 con 0x7f09dc0c71d0 2024-09-13T16:44:03.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7f09e2a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc06d230 con 0x7f09dc0c75b0 2024-09-13T16:44:03.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.078+0000 7f7882a00640 1 -- 172.21.15.28:0/2990707434 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f788c103270 con 0x7f78840e6610 2024-09-13T16:44:03.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.078+0000 7f7882a00640 1 --2- 172.21.15.28:0/2990707434 >> [v2:172.21.15.28:6816/886887303,v1:172.21.15.28:6817/886887303] conn(0x7f786c072e00 0x7f786c075220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.078+0000 7f7882a00640 1 -- 172.21.15.28:0/2990707434 --> [v2:172.21.15.28:6816/886887303,v1:172.21.15.28:6817/886887303] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f786c0758d0 con 0x7f786c072e00 2024-09-13T16:44:03.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.078+0000 7f7882a00640 1 -- 172.21.15.28:0/2990707434 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f788c1036c0 con 0x7f78840e6610 2024-09-13T16:44:03.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7f09dbe00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f09dc0c71d0 0x7f09dc06dc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:51510/0 (socket says 172.21.15.28:51510) 2024-09-13T16:44:03.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7f09dbe00640 1 -- 172.21.15.28:0/1860061131 learned_addr learned my addr 172.21.15.28:0/1860061131 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:03.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7f89bcc00640 1 --2- 172.21.15.28:0/1085338538 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f89a006ac60 0x7f89a006d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7fb237e00640 1 --2- 172.21.15.28:0/828383813 >> [v2:172.21.15.28:6808/23768570,v1:172.21.15.28:6809/23768570] conn(0x7fb21807b6c0 0x7fb21807dae0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7fa413200640 1 -- 172.21.15.28:0/4242220132 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40c15c150 msgr2=0x7fa40c15c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7fa413200640 1 --2- 172.21.15.28:0/4242220132 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40c15c150 0x7fa40c15c580 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa404009e80 tx=0x7fa40402a6f0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7fa413200640 1 -- 172.21.15.28:0/4242220132 shutdown_connections 2024-09-13T16:44:03.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7fa413200640 1 --2- 172.21.15.28:0/4242220132 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40c15c150 0x7fa40c15c580 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7fa413200640 1 -- 172.21.15.28:0/4242220132 >> 172.21.15.28:0/4242220132 conn(0x7fa40c065c50 msgr2=0x7fa40c066060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7fe25ea00640 1 -- 172.21.15.28:0/538826190 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c032440 con 0x7fe260071950 2024-09-13T16:44:03.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7fe25ea00640 1 -- 172.21.15.28:0/538826190 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24c0325a0 con 0x7fe260071950 2024-09-13T16:44:03.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7fe25ea00640 1 -- 172.21.15.28:0/538826190 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c032930 con 0x7fe260071950 2024-09-13T16:44:03.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7f599d600640 1 --2- 172.21.15.28:0/3203222142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f599806df90 0x7f59980c5ac0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f599806ca00 tx=0x7f599002a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2861f5f9cd307357 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7f4e63200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4e641571a0 0x7f4e640c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7f4e63200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4e640c4fa0 0x7f4e6415cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7f4e63200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e64157580 con 0x7f4e64160ee0 2024-09-13T16:44:03.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7f4e63200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e640c1a90 con 0x7f4e640c4fa0 2024-09-13T16:44:03.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7f4e63200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e6415d330 con 0x7f4e641571a0 2024-09-13T16:44:03.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7f89bcc00640 1 -- 172.21.15.28:0/1085338538 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89ac088a00 con 0x7f89b80d8830 2024-09-13T16:44:03.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7f89c4c00640 1 --2- 172.21.15.28:0/1085338538 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f89a006ac60 0x7f89a006d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7f89bcc00640 1 --2- 172.21.15.28:0/1085338538 >> [v2:172.21.15.28:6800/2964323435,v1:172.21.15.28:6801/2964323435] conn(0x7f89a0072e80 0x7f89a00752a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7f89bcc00640 1 -- 172.21.15.28:0/1085338538 --> [v2:172.21.15.28:6800/2964323435,v1:172.21.15.28:6801/2964323435] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f89a0075950 con 0x7f89a0072e80 2024-09-13T16:44:03.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7f89bcc00640 1 -- 172.21.15.28:0/1085338538 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89ac088db0 con 0x7f89b80d8830 2024-09-13T16:44:03.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.079+0000 7f09d3400640 1 --2- 172.21.15.28:0/1860061131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc0c75b0 0x7f09dc1580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7f5997400640 1 -- 172.21.15.28:0/3203222142 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5990007540 con 0x7f599806df90 2024-09-13T16:44:03.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.081+0000 7fb22f400640 1 -- 172.21.15.28:0/828383813 <== osd.1 v2:172.21.15.28:6808/23768570 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fb21807e190 con 0x7fb21807b6c0 2024-09-13T16:44:03.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7f788a800640 1 --2- 172.21.15.28:0/2990707434 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f786c06ac60 0x7f786c06d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7f7889e00640 1 --2- 172.21.15.28:0/2990707434 >> [v2:172.21.15.28:6816/886887303,v1:172.21.15.28:6817/886887303] conn(0x7f786c072e00 0x7f786c075220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.081+0000 7fa413200640 1 -- 172.21.15.28:0/4242220132 shutdown_connections 2024-09-13T16:44:03.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.081+0000 7fa413200640 1 -- 172.21.15.28:0/4242220132 wait complete. 2024-09-13T16:44:03.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7fe266a00640 1 -- 172.21.15.28:0/538826190 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe260071950 msgr2=0x7fe2600c6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7fe266a00640 1 --2- 172.21.15.28:0/538826190 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe260071950 0x7fe2600c6f60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe26008ac00 tx=0x7fe24c02a1e0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.081+0000 7fe266a00640 1 -- 172.21.15.28:0/538826190 shutdown_connections 2024-09-13T16:44:03.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.081+0000 7fe266a00640 1 --2- 172.21.15.28:0/538826190 >> v2:172.21.15.191:3300/0 conn(0x7fe2600c74a0 0x7fe26006d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.081+0000 7fe266a00640 1 --2- 172.21.15.28:0/538826190 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe260071950 0x7fe2600c6f60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.081+0000 7fe266a00640 1 --2- 172.21.15.28:0/538826190 >> v2:172.21.15.28:3300/0 conn(0x7fe2600c25c0 0x7fe260071410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.081+0000 7fe266a00640 1 -- 172.21.15.28:0/538826190 >> 172.21.15.28:0/538826190 conn(0x7fe2600ca6b0 msgr2=0x7fe2600caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7f09db400640 1 -- 172.21.15.28:0/1860061131 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174418110 0 0) 0x7f09dc158810 con 0x7f09dc0c71d0 2024-09-13T16:44:03.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7f09d3400640 1 -- 172.21.15.28:0/1860061131 >> v1:172.21.15.191:6789/0 conn(0x7f09dc0c71d0 legacy=0x7f09dc06dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7f09d3400640 1 -- 172.21.15.28:0/1860061131 >> v1:172.21.15.28:6789/0 conn(0x7f09dc0c65f0 legacy=0x7f09dc14e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7f09d3400640 1 -- 172.21.15.28:0/1860061131 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09dc158af0 con 0x7f09dc0c75b0 2024-09-13T16:44:03.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.081+0000 7f5997400640 1 -- 172.21.15.28:0/3203222142 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599002d050 con 0x7f599806df90 2024-09-13T16:44:03.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.081+0000 7f5997400640 1 -- 172.21.15.28:0/3203222142 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59900078c0 con 0x7f599806df90 2024-09-13T16:44:03.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7f4e62800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4e64160ee0 0x7f4e640c6cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59526/0 (socket says 172.21.15.28:59526) 2024-09-13T16:44:03.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7f4e62800640 1 -- 172.21.15.28:0/2571701736 learned_addr learned my addr 172.21.15.28:0/2571701736 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:03.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.080+0000 7f4e61400640 1 --2- 172.21.15.28:0/2571701736 >> v2:172.21.15.28:3300/0 conn(0x7f4e640c4fa0 0x7f4e6415cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.081+0000 7f4e61e00640 1 --2- 172.21.15.28:0/2571701736 >> v2:172.21.15.191:3300/0 conn(0x7f4e641571a0 0x7f4e640c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.081+0000 7f4e61400640 1 -- 172.21.15.28:0/2571701736 >> v2:172.21.15.191:3300/0 conn(0x7f4e641571a0 msgr2=0x7f4e640c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.081+0000 7f4e61400640 1 --2- 172.21.15.28:0/2571701736 >> v2:172.21.15.191:3300/0 conn(0x7f4e641571a0 0x7f4e640c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.081+0000 7f4e61400640 1 -- 172.21.15.28:0/2571701736 >> v1:172.21.15.28:6789/0 conn(0x7f4e64160ee0 legacy=0x7f4e640c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.081+0000 7f4e61400640 1 -- 172.21.15.28:0/2571701736 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6406fdd0 con 0x7f4e640c4fa0 2024-09-13T16:44:03.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.081+0000 7f4e61e00640 1 --2- 172.21.15.28:0/2571701736 >> v2:172.21.15.191:3300/0 conn(0x7f4e641571a0 0x7f4e640c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:03.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.082+0000 7f7889e00640 1 --2- 172.21.15.28:0/2990707434 >> [v2:172.21.15.28:6816/886887303,v1:172.21.15.28:6817/886887303] conn(0x7f786c072e00 0x7f786c075220 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.082+0000 7f788a800640 1 --2- 172.21.15.28:0/2990707434 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f786c06ac60 0x7f786c06d120 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f787c034000 tx=0x7f787c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.082+0000 7f09d3400640 1 --2- 172.21.15.28:0/1860061131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc0c75b0 0x7f09dc1580b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f09c40097d0 tx=0x7f09c402a2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78c73967dbf89432 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.082+0000 7f599f600640 1 -- 172.21.15.28:0/3203222142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f599806df90 msgr2=0x7f59980c5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.082+0000 7f599f600640 1 --2- 172.21.15.28:0/3203222142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f599806df90 0x7f59980c5ac0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f599806ca00 tx=0x7f599002a620 comp rx=0 tx=0).stop 2024-09-13T16:44:03.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.082+0000 7fe266a00640 1 -- 172.21.15.28:0/538826190 shutdown_connections 2024-09-13T16:44:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.082+0000 7f09db400640 1 -- 172.21.15.28:0/1860061131 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c4032640 con 0x7f09dc0c75b0 2024-09-13T16:44:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.081+0000 7f89c5600640 1 --2- 172.21.15.28:0/1085338538 >> [v2:172.21.15.28:6800/2964323435,v1:172.21.15.28:6801/2964323435] conn(0x7f89a0072e80 0x7f89a00752a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.082+0000 7f89c4c00640 1 --2- 172.21.15.28:0/1085338538 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f89a006ac60 0x7f89a006d120 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f89b4034000 tx=0x7f89b402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.082+0000 7f89c5600640 1 --2- 172.21.15.28:0/1085338538 >> [v2:172.21.15.28:6800/2964323435,v1:172.21.15.28:6801/2964323435] conn(0x7f89a0072e80 0x7f89a00752a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.083+0000 7f09db400640 1 -- 172.21.15.28:0/1860061131 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c40327a0 con 0x7f09dc0c75b0 2024-09-13T16:44:03.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.083+0000 7f4e61400640 1 --2- 172.21.15.28:0/2571701736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e640c4fa0 0x7f4e6415cdf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4e640cb360 tx=0x7f4e5c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f53dc4e5820a2e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.084+0000 7f599f600640 1 -- 172.21.15.28:0/3203222142 shutdown_connections 2024-09-13T16:44:03.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.083+0000 7f7882a00640 1 -- 172.21.15.28:0/2990707434 <== osd.2 v2:172.21.15.28:6816/886887303 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f786c0758d0 con 0x7f786c072e00 2024-09-13T16:44:03.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.084+0000 7f09e2a00640 1 -- 172.21.15.28:0/1860061131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc0c75b0 msgr2=0x7f09dc1580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.084+0000 7f09e2a00640 1 --2- 172.21.15.28:0/1860061131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc0c75b0 0x7f09dc1580b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f09c40097d0 tx=0x7f09c402a2c0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.083+0000 7f4e60a00640 1 -- 172.21.15.28:0/2571701736 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c007740 con 0x7f4e640c4fa0 2024-09-13T16:44:03.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.084+0000 7f4e60a00640 1 -- 172.21.15.28:0/2571701736 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5c02d050 con 0x7f4e640c4fa0 2024-09-13T16:44:03.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.084+0000 7f4e60a00640 1 -- 172.21.15.28:0/2571701736 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c007b00 con 0x7f4e640c4fa0 2024-09-13T16:44:03.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.084+0000 7f599f600640 1 --2- 172.21.15.28:0/3203222142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f599806df90 0x7f59980c5ac0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.084+0000 7f599f600640 1 --2- 172.21.15.28:0/3203222142 >> v2:172.21.15.191:3300/0 conn(0x7f5998157110 0x7f59980c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.084+0000 7f599f600640 1 -- 172.21.15.28:0/3203222142 >> 172.21.15.28:0/3203222142 conn(0x7f59980ca6b0 msgr2=0x7f59980caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.084+0000 7f09e2a00640 1 -- 172.21.15.28:0/1860061131 shutdown_connections 2024-09-13T16:44:03.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.084+0000 7f599f600640 1 -- 172.21.15.28:0/3203222142 shutdown_connections 2024-09-13T16:44:03.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.084+0000 7fa413200640 1 Processor -- start 2024-09-13T16:44:03.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.084+0000 7f89bcc00640 1 -- 172.21.15.28:0/1085338538 <== osd.0 v2:172.21.15.28:6800/2964323435 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f89a0075950 con 0x7f89a0072e80 2024-09-13T16:44:03.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.084+0000 7f09e2a00640 1 --2- 172.21.15.28:0/1860061131 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc0c75b0 0x7f09dc1580b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.084+0000 7f09e2a00640 1 -- 172.21.15.28:0/1860061131 >> 172.21.15.28:0/1860061131 conn(0x7f09dc0ca6b0 msgr2=0x7f09dc0caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.082+0000 7fe266a00640 1 -- 172.21.15.28:0/538826190 wait complete. 2024-09-13T16:44:03.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.085+0000 7f4e63200640 1 -- 172.21.15.28:0/2571701736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e640c4fa0 msgr2=0x7f4e6415cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.085+0000 7f4e63200640 1 --2- 172.21.15.28:0/2571701736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e640c4fa0 0x7f4e6415cdf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4e640cb360 tx=0x7f4e5c02a6a0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.084+0000 7fa413200640 1 -- start start 2024-09-13T16:44:03.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.084+0000 7f599f600640 1 -- 172.21.15.28:0/3203222142 wait complete. 2024-09-13T16:44:03.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.085+0000 7fe266a00640 1 Processor -- start 2024-09-13T16:44:03.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.085+0000 7fe266a00640 1 -- start start 2024-09-13T16:44:03.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.086+0000 7f4e63200640 1 -- 172.21.15.28:0/2571701736 shutdown_connections 2024-09-13T16:44:03.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.086+0000 7f4e63200640 1 --2- 172.21.15.28:0/2571701736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e640c4fa0 0x7f4e6415cdf0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.086+0000 7f4e63200640 1 --2- 172.21.15.28:0/2571701736 >> v2:172.21.15.191:3300/0 conn(0x7f4e641571a0 0x7f4e640c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.086+0000 7f4e63200640 1 -- 172.21.15.28:0/2571701736 >> 172.21.15.28:0/2571701736 conn(0x7f4e64069e80 msgr2=0x7f4e6406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.086+0000 7fa413200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40c0ca5d0 0x7fa40c14d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.086+0000 7fa413200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa40c152f20 0x7fa40c1e3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.086+0000 7fa413200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa40c15c150 0x7fa40c1f5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.086+0000 7fa413200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40c14d700 con 0x7fa40c15c150 2024-09-13T16:44:03.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.086+0000 7fa413200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40c0b1030 con 0x7fa40c152f20 2024-09-13T16:44:03.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.086+0000 7fa413200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa40c150050 con 0x7fa40c0ca5d0 2024-09-13T16:44:03.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.086+0000 7fa40be00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40c0ca5d0 0x7fa40c14d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.086+0000 7fa40be00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40c0ca5d0 0x7fa40c14d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37396/0 (socket says 172.21.15.28:37396) 2024-09-13T16:44:03.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.086+0000 7fa40be00640 1 -- 172.21.15.28:0/3473069070 learned_addr learned my addr 172.21.15.28:0/3473069070 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.087+0000 7fa40b400640 1 --2- 172.21.15.28:0/3473069070 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa40c152f20 0x7fa40c1e3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.085+0000 7f09e2a00640 1 -- 172.21.15.28:0/1860061131 shutdown_connections 2024-09-13T16:44:03.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.085+0000 7f09e2a00640 1 -- 172.21.15.28:0/1860061131 wait complete. 2024-09-13T16:44:03.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.087+0000 7f4e63200640 1 -- 172.21.15.28:0/2571701736 shutdown_connections 2024-09-13T16:44:03.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.087+0000 7fa411200640 1 --2- 172.21.15.28:0/3473069070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa40c15c150 0x7fa40c1f5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.087+0000 7fe266a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe260071950 0x7fe260176b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.087+0000 7fa40b400640 1 -- 172.21.15.28:0/3473069070 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40c0ca5d0 msgr2=0x7fa40c14d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.087+0000 7fa40b400640 1 --2- 172.21.15.28:0/3473069070 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa40c0ca5d0 0x7fa40c14d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.087+0000 7fa40b400640 1 -- 172.21.15.28:0/3473069070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa40c15c150 msgr2=0x7fa40c1f5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.087+0000 7fe266a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2600c25c0 0x7fe26020cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.087+0000 7f599f600640 1 Processor -- start 2024-09-13T16:44:03.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7f599f600640 1 -- start start 2024-09-13T16:44:03.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.087+0000 7f09e2a00640 1 Processor -- start 2024-09-13T16:44:03.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7f09e2a00640 1 -- start start 2024-09-13T16:44:03.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7fa40b400640 1 --2- 172.21.15.28:0/3473069070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa40c15c150 0x7fa40c1f5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7fe25fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe260071950 0x7fe260176b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7fe266a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe2600c74a0 0x7fe2601758c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7fa40b400640 1 -- 172.21.15.28:0/3473069070 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa404009b30 con 0x7fa40c152f20 2024-09-13T16:44:03.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7fa40b400640 1 --2- 172.21.15.28:0/3473069070 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa40c152f20 0x7fa40c1e3d80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800c410 tx=0x7fa3f800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7fa411200640 1 --2- 172.21.15.28:0/3473069070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa40c15c150 0x7fa40c1f5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:03.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7fa403e00640 1 -- 172.21.15.28:0/3473069070 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8005230 con 0x7fa40c152f20 2024-09-13T16:44:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7fa403e00640 1 -- 172.21.15.28:0/3473069070 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f80053b0 con 0x7fa40c152f20 2024-09-13T16:44:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7fa403e00640 1 -- 172.21.15.28:0/3473069070 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8004820 con 0x7fa40c152f20 2024-09-13T16:44:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7fe266a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2601596b0 con 0x7fe260071950 2024-09-13T16:44:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7fe266a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26020c770 con 0x7fe2600c25c0 2024-09-13T16:44:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7fe266a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26006a640 con 0x7fe2600c74a0 2024-09-13T16:44:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7fe25f400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2600c25c0 0x7fe26020cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7fe25f400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2600c25c0 0x7fe26020cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60654/0 (socket says 172.21.15.28:60654) 2024-09-13T16:44:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7fe25f400640 1 -- 172.21.15.28:0/1813373032 learned_addr learned my addr 172.21.15.28:0/1813373032 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.088+0000 7fe264a00640 1 --2- 172.21.15.28:0/1813373032 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe2600c74a0 0x7fe2601758c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.089+0000 7fe25f400640 1 -- 172.21.15.28:0/1813373032 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe2600c74a0 msgr2=0x7fe2601758c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.089+0000 7fe25f400640 1 --2- 172.21.15.28:0/1813373032 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe2600c74a0 0x7fe2601758c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.089+0000 7fe25f400640 1 -- 172.21.15.28:0/1813373032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe260071950 msgr2=0x7fe260176b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.089+0000 7f09e2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09dc0c65f0 0x7f09dc071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.089+0000 7f09e2a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09dc0c71d0 0x7f09dc1461f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.089+0000 7f09e2a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc0c75b0 0x7f09dc155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.089+0000 7f09e2a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc157690 con 0x7f09dc0c65f0 2024-09-13T16:44:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.089+0000 7f09e2a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc155a50 con 0x7f09dc0c71d0 2024-09-13T16:44:03.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.089+0000 7f09e2a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc1452e0 con 0x7f09dc0c75b0 2024-09-13T16:44:03.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.089+0000 7fe25f400640 1 --2- 172.21.15.28:0/1813373032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe260071950 0x7fe260176b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.087+0000 7f4e63200640 1 -- 172.21.15.28:0/2571701736 wait complete. 2024-09-13T16:44:03.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.089+0000 7fa413200640 1 -- 172.21.15.28:0/3473069070 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa40c14c7a0 con 0x7fa40c152f20 2024-09-13T16:44:03.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.089+0000 7f4e63200640 1 Processor -- start 2024-09-13T16:44:03.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.090+0000 7fe25f400640 1 -- 172.21.15.28:0/1813373032 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24c0094d0 con 0x7fe2600c25c0 2024-09-13T16:44:03.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.090+0000 7fa413200640 1 -- 172.21.15.28:0/3473069070 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa40c0c9020 con 0x7fa40c152f20 2024-09-13T16:44:03.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.090+0000 7fe25fe00640 1 --2- 172.21.15.28:0/1813373032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe260071950 0x7fe260176b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:44:03.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.090+0000 7f599f600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f599806df90 0x7f599815e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.090+0000 7f599f600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5998157110 0x7f59981f2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.090+0000 7f599f600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59981574f0 0x7f599815ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.090+0000 7f599f600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5998158620 con 0x7f5998157110 2024-09-13T16:44:03.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.090+0000 7f09dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09dc0c65f0 0x7f09dc071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.090+0000 7f09dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09dc0c65f0 0x7f09dc071040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41528/0 (socket says 172.21.15.28:41528) 2024-09-13T16:44:03.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.090+0000 7f09dbe00640 1 -- 172.21.15.28:0/406124455 learned_addr learned my addr 172.21.15.28:0/406124455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.090+0000 7f4e63200640 1 -- start start 2024-09-13T16:44:03.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.090+0000 7f599f600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599815fcf0 con 0x7f599806df90 2024-09-13T16:44:03.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.090+0000 7f599f600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5998160550 con 0x7f59981574f0 2024-09-13T16:44:03.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.090+0000 7f09e0a00640 1 --2- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc0c75b0 0x7f09dc155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.090+0000 7f5997e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5998157110 0x7f59981f2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.091+0000 7f5997e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5998157110 0x7f59981f2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41532/0 (socket says 172.21.15.28:41532) 2024-09-13T16:44:03.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.091+0000 7f5997e00640 1 -- 172.21.15.28:0/3389780006 learned_addr learned my addr 172.21.15.28:0/3389780006 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.091+0000 7f599cc00640 1 --2- 172.21.15.28:0/3389780006 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f599806df90 0x7f599815e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.091+0000 7f599d600640 1 --2- 172.21.15.28:0/3389780006 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59981574f0 0x7f599815ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.091+0000 7f09d3400640 1 --2- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09dc0c71d0 0x7f09dc1461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.091+0000 7f09dbe00640 1 -- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc0c75b0 msgr2=0x7f09dc155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.091+0000 7fe25f400640 1 --2- 172.21.15.28:0/1813373032 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2600c25c0 0x7fe26020cc00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe24c000c00 tx=0x7fe24c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.091+0000 7f09dbe00640 1 --2- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc0c75b0 0x7f09dc155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.092+0000 7fe257e00640 1 -- 172.21.15.28:0/1813373032 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c032440 con 0x7fe2600c25c0 2024-09-13T16:44:03.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.092+0000 7fe257e00640 1 -- 172.21.15.28:0/1813373032 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24c030b80 con 0x7fe2600c25c0 2024-09-13T16:44:03.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.092+0000 7fe257e00640 1 -- 172.21.15.28:0/1813373032 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c03d440 con 0x7fe2600c25c0 2024-09-13T16:44:03.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.092+0000 7fe266a00640 1 -- 172.21.15.28:0/1813373032 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2601590f0 con 0x7fe2600c25c0 2024-09-13T16:44:03.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.092+0000 7f5997e00640 1 -- 172.21.15.28:0/3389780006 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59981574f0 msgr2=0x7f599815ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.092+0000 7f5997e00640 1 --2- 172.21.15.28:0/3389780006 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f59981574f0 0x7f599815ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.092+0000 7f5997e00640 1 -- 172.21.15.28:0/3389780006 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f599806df90 msgr2=0x7f599815e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.092+0000 7f5997e00640 1 --2- 172.21.15.28:0/3389780006 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f599806df90 0x7f599815e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.092+0000 7f5997e00640 1 -- 172.21.15.28:0/3389780006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5990009a70 con 0x7f5998157110 2024-09-13T16:44:03.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.092+0000 7f09dbe00640 1 -- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09dc0c71d0 msgr2=0x7f09dc1461f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.092+0000 7f09dbe00640 1 --2- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09dc0c71d0 0x7f09dc1461f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.092+0000 7f09dbe00640 1 -- 172.21.15.28:0/406124455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c4009400 con 0x7f09dc0c65f0 2024-09-13T16:44:03.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.092+0000 7f09d3400640 1 --2- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09dc0c71d0 0x7f09dc1461f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:44:03.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.093+0000 7f09e0a00640 1 --2- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09dc0c75b0 0x7f09dc155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:03.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.093+0000 7f4e63200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e640c4fa0 0x7f4e641527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.093+0000 7fa413200640 1 -- 172.21.15.28:0/3473069070 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa40c06b5f0 con 0x7fa40c152f20 2024-09-13T16:44:03.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.093+0000 7fa403e00640 1 -- 172.21.15.28:0/3473069070 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa3f8004980 con 0x7fa40c152f20 2024-09-13T16:44:03.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.093+0000 7f5997e00640 1 --2- 172.21.15.28:0/3389780006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5998157110 0x7f59981f2f10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f598400c430 tx=0x7f598400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.093+0000 7f4e63200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e641571a0 0x7f4e641f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.093+0000 7f4e63200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e64160ee0 0x7f4e641e6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.093+0000 7f4e63200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e64152d20 con 0x7f4e640c4fa0 2024-09-13T16:44:03.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.093+0000 7f4e61400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e641571a0 0x7f4e641f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.092+0000 7fe266a00640 1 -- 172.21.15.28:0/1813373032 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26020c400 con 0x7fe2600c25c0 2024-09-13T16:44:03.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.093+0000 7fe266a00640 1 -- 172.21.15.28:0/1813373032 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe26015ceb0 con 0x7fe2600c25c0 2024-09-13T16:44:03.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.094+0000 7f4e61400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e641571a0 0x7f4e641f5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37442/0 (socket says 172.21.15.28:37442) 2024-09-13T16:44:03.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.094+0000 7f4e61400640 1 -- 172.21.15.28:0/3335021308 learned_addr learned my addr 172.21.15.28:0/3335021308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.094+0000 7f4e62800640 1 --2- 172.21.15.28:0/3335021308 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e64160ee0 0x7f4e641e6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.094+0000 7f09dbe00640 1 --2- 172.21.15.28:0/406124455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09dc0c65f0 0x7f09dc071040 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00c4a0 tx=0x7f09cc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.094+0000 7f4e61e00640 1 --2- 172.21.15.28:0/3335021308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e640c4fa0 0x7f4e641527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.094+0000 7f4e63200640 1 -- 172.21.15.28:0/3335021308 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e64151c00 con 0x7f4e64160ee0 2024-09-13T16:44:03.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.094+0000 7f4e63200640 1 -- 172.21.15.28:0/3335021308 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e641f69a0 con 0x7f4e641571a0 2024-09-13T16:44:03.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.094+0000 7f09d8c00640 1 -- 172.21.15.28:0/406124455 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc0074e0 con 0x7f09dc0c65f0 2024-09-13T16:44:03.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.094+0000 7f09d8c00640 1 -- 172.21.15.28:0/406124455 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09cc007640 con 0x7f09dc0c65f0 2024-09-13T16:44:03.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.094+0000 7f09e2a00640 1 -- 172.21.15.28:0/406124455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09dc147d90 con 0x7f09dc0c65f0 2024-09-13T16:44:03.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.094+0000 7f09d8c00640 1 -- 172.21.15.28:0/406124455 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc004a00 con 0x7f09dc0c65f0 2024-09-13T16:44:03.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.094+0000 7f4e62800640 1 -- 172.21.15.28:0/3335021308 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e641571a0 msgr2=0x7f4e641f5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.094+0000 7f4e62800640 1 --2- 172.21.15.28:0/3335021308 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e641571a0 0x7f4e641f5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.094+0000 7f4e62800640 1 -- 172.21.15.28:0/3335021308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e640c4fa0 msgr2=0x7f4e641527e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.094+0000 7f4e62800640 1 --2- 172.21.15.28:0/3335021308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e640c4fa0 0x7f4e641527e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.094+0000 7f4e62800640 1 -- 172.21.15.28:0/3335021308 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e54009400 con 0x7f4e64160ee0 2024-09-13T16:44:03.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.095+0000 7fa403e00640 1 --2- 172.21.15.28:0/3473069070 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa3ec06ac50 0x7fa3ec06d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.095+0000 7f09e2a00640 1 -- 172.21.15.28:0/406124455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09dc1455a0 con 0x7f09dc0c65f0 2024-09-13T16:44:03.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.095+0000 7f4e62800640 1 --2- 172.21.15.28:0/3335021308 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e64160ee0 0x7f4e641e6240 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4e5000c3c0 tx=0x7f4e5000c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.095+0000 7f4e61400640 1 --2- 172.21.15.28:0/3335021308 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e641571a0 0x7f4e641f5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:03.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.095+0000 7fa403e00640 1 -- 172.21.15.28:0/3473069070 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3f8088800 con 0x7fa40c152f20 2024-09-13T16:44:03.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.095+0000 7fa40be00640 1 --2- 172.21.15.28:0/3473069070 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa3ec06ac50 0x7fa3ec06d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.096+0000 7f5994c00640 1 -- 172.21.15.28:0/3389780006 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984011790 con 0x7f5998157110 2024-09-13T16:44:03.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.096+0000 7f5994c00640 1 -- 172.21.15.28:0/3389780006 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59840118f0 con 0x7f5998157110 2024-09-13T16:44:03.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.096+0000 7f5994c00640 1 -- 172.21.15.28:0/3389780006 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984011ca0 con 0x7f5998157110 2024-09-13T16:44:03.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.095+0000 7f4e5a000640 1 -- 172.21.15.28:0/3335021308 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e50005ce0 con 0x7f4e64160ee0 2024-09-13T16:44:03.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.096+0000 7f4e63200640 1 -- 172.21.15.28:0/3335021308 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e5c009a70 con 0x7f4e64160ee0 2024-09-13T16:44:03.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.096+0000 7f4e63200640 1 -- 172.21.15.28:0/3335021308 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e64150d90 con 0x7f4e64160ee0 2024-09-13T16:44:03.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.097+0000 7f599f600640 1 -- 172.21.15.28:0/3389780006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59981f3450 con 0x7f5998157110 2024-09-13T16:44:03.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.097+0000 7f599f600640 1 -- 172.21.15.28:0/3389780006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f599815f050 con 0x7f5998157110 2024-09-13T16:44:03.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.097+0000 7f4e5a000640 1 -- 172.21.15.28:0/3335021308 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e50005e40 con 0x7f4e64160ee0 2024-09-13T16:44:03.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.095+0000 7fa403e00640 1 --2- 172.21.15.28:0/3473069070 >> [v2:172.21.15.28:6824/2616036674,v1:172.21.15.28:6825/2616036674] conn(0x7fa3ec072df0 0x7fa3ec075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.096+0000 7fa403e00640 1 -- 172.21.15.28:0/3473069070 --> [v2:172.21.15.28:6824/2616036674,v1:172.21.15.28:6825/2616036674] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3ec0758c0 con 0x7fa3ec072df0 2024-09-13T16:44:03.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.096+0000 7fa403e00640 1 -- 172.21.15.28:0/3473069070 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3f8053910 con 0x7fa40c152f20 2024-09-13T16:44:03.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.098+0000 7f09d8c00640 1 -- 172.21.15.28:0/406124455 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f09cc004b60 con 0x7f09dc0c65f0 2024-09-13T16:44:03.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.098+0000 7f4e63200640 1 -- 172.21.15.28:0/3335021308 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4e3c000fc0 con 0x7f4e64160ee0 2024-09-13T16:44:03.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.098+0000 7f4e5a000640 1 -- 172.21.15.28:0/3335021308 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e50005ce0 con 0x7f4e64160ee0 2024-09-13T16:44:03.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.098+0000 7fa40be00640 1 --2- 172.21.15.28:0/3473069070 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa3ec06ac50 0x7fa3ec06d110 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0018b0 tx=0x7fa3fc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.098+0000 7fa411200640 1 --2- 172.21.15.28:0/3473069070 >> [v2:172.21.15.28:6824/2616036674,v1:172.21.15.28:6825/2616036674] conn(0x7fa3ec072df0 0x7fa3ec075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.099+0000 7f09d8c00640 1 --2- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f09b806ac30 0x7f09b806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.099+0000 7f09d8c00640 1 -- 172.21.15.28:0/406124455 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09cc0884b0 con 0x7f09dc0c65f0 2024-09-13T16:44:03.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.099+0000 7f09d3400640 1 --2- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f09b806ac30 0x7f09b806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.100+0000 7f599f600640 1 -- 172.21.15.28:0/3389780006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f599815d640 con 0x7f5998157110 2024-09-13T16:44:03.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.100+0000 7f09e2a00640 1 --2- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:6808/3006595088,v1:172.21.15.191:6809/3006595088] conn(0x7f09b40015e0 0x7f09b4003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.100+0000 7f09e2a00640 1 -- 172.21.15.28:0/406124455 --> [v2:172.21.15.191:6808/3006595088,v1:172.21.15.191:6809/3006595088] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09b4006b40 con 0x7f09b40015e0 2024-09-13T16:44:03.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.100+0000 7f09d3400640 1 --2- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f09b806ac30 0x7f09b806d0f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f09c4034000 tx=0x7f09c402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:44:03.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.101+0000 7fa411200640 1 --2- 172.21.15.28:0/3473069070 >> [v2:172.21.15.28:6824/2616036674,v1:172.21.15.28:6825/2616036674] conn(0x7fa3ec072df0 0x7fa3ec075210 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.100+0000 7f09e0a00640 1 --2- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:6808/3006595088,v1:172.21.15.191:6809/3006595088] conn(0x7f09b40015e0 0x7f09b4003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.101+0000 7f09e0a00640 1 --2- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:6808/3006595088,v1:172.21.15.191:6809/3006595088] conn(0x7f09b40015e0 0x7f09b4003aa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.101+0000 7f5994c00640 1 -- 172.21.15.28:0/3389780006 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f598401c460 con 0x7f5998157110 2024-09-13T16:44:03.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.101+0000 7f4e5a000640 1 -- 172.21.15.28:0/3335021308 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4e500070d0 con 0x7f4e64160ee0 2024-09-13T16:44:03.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.098+0000 7fe257e00640 1 -- 172.21.15.28:0/1813373032 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe24c039040 con 0x7fe2600c25c0 2024-09-13T16:44:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.101+0000 7fe257e00640 1 --2- 172.21.15.28:0/1813373032 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe24006acd0 0x7fe24006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.101+0000 7fe257e00640 1 -- 172.21.15.28:0/1813373032 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe24c0a9d20 con 0x7fe2600c25c0 2024-09-13T16:44:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.102+0000 7fe257e00640 1 --2- 172.21.15.28:0/1813373032 >> [v2:172.21.15.191:6800/2817102450,v1:172.21.15.191:6801/2817102450] conn(0x7fe240072e70 0x7fe240075290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.102+0000 7fe257e00640 1 -- 172.21.15.28:0/1813373032 --> [v2:172.21.15.191:6800/2817102450,v1:172.21.15.191:6801/2817102450] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe240075940 con 0x7fe240072e70 2024-09-13T16:44:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.102+0000 7fe257e00640 1 -- 172.21.15.28:0/1813373032 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe24c074f70 con 0x7fe2600c25c0 2024-09-13T16:44:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.102+0000 7fa403e00640 1 -- 172.21.15.28:0/3473069070 <== osd.3 v2:172.21.15.28:6824/2616036674 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fa3ec0758c0 con 0x7fa3ec072df0 2024-09-13T16:44:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.102+0000 7f4e5a000640 1 --2- 172.21.15.28:0/3335021308 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4e4406ac60 0x7f4e4406d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.102+0000 7f4e5a000640 1 -- 172.21.15.28:0/3335021308 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e500885f0 con 0x7f4e64160ee0 2024-09-13T16:44:03.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.102+0000 7f09d8c00640 1 -- 172.21.15.28:0/406124455 <== osd.6 v2:172.21.15.191:6808/3006595088 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f09b4006b40 con 0x7f09b40015e0 2024-09-13T16:44:03.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.103+0000 7f4e5a000640 1 --2- 172.21.15.28:0/3335021308 >> [v2:172.21.15.191:6824/1477327720,v1:172.21.15.191:6825/1477327720] conn(0x7f4e44072e00 0x7f4e44075220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.103+0000 7fe25fe00640 1 --2- 172.21.15.28:0/1813373032 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe24006acd0 0x7fe24006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.103+0000 7fe264a00640 1 --2- 172.21.15.28:0/1813373032 >> [v2:172.21.15.191:6800/2817102450,v1:172.21.15.191:6801/2817102450] conn(0x7fe240072e70 0x7fe240075290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.103+0000 7f4e5a000640 1 -- 172.21.15.28:0/3335021308 --> [v2:172.21.15.191:6824/1477327720,v1:172.21.15.191:6825/1477327720] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e440758d0 con 0x7f4e44072e00 2024-09-13T16:44:03.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.103+0000 7f4e5a000640 1 -- 172.21.15.28:0/3335021308 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e50088a40 con 0x7f4e64160ee0 2024-09-13T16:44:03.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.103+0000 7f4e61e00640 1 --2- 172.21.15.28:0/3335021308 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4e4406ac60 0x7f4e4406d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.103+0000 7f4e61400640 1 --2- 172.21.15.28:0/3335021308 >> [v2:172.21.15.191:6824/1477327720,v1:172.21.15.191:6825/1477327720] conn(0x7f4e44072e00 0x7f4e44075220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.104+0000 7f5994c00640 1 --2- 172.21.15.28:0/3389780006 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f597806ac50 0x7f597806d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.104+0000 7f4e61400640 1 --2- 172.21.15.28:0/3335021308 >> [v2:172.21.15.191:6824/1477327720,v1:172.21.15.191:6825/1477327720] conn(0x7f4e44072e00 0x7f4e44075220 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.104+0000 7f5994c00640 1 -- 172.21.15.28:0/3389780006 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5984089030 con 0x7f5998157110 2024-09-13T16:44:03.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.104+0000 7f599cc00640 1 --2- 172.21.15.28:0/3389780006 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f597806ac50 0x7f597806d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.104+0000 7fe264a00640 1 --2- 172.21.15.28:0/1813373032 >> [v2:172.21.15.191:6800/2817102450,v1:172.21.15.191:6801/2817102450] conn(0x7fe240072e70 0x7fe240075290 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.105+0000 7f4e61e00640 1 --2- 172.21.15.28:0/3335021308 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4e4406ac60 0x7f4e4406d120 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4e54009c30 tx=0x7f4e54009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.105+0000 7fe25fe00640 1 --2- 172.21.15.28:0/1813373032 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe24006acd0 0x7fe24006d190 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe250006fd0 tx=0x7fe250008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.105+0000 7f4e5a000640 1 -- 172.21.15.28:0/3335021308 <== osd.7 v2:172.21.15.191:6824/1477327720 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f4e440758d0 con 0x7f4e44072e00 2024-09-13T16:44:03.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.105+0000 7f599cc00640 1 --2- 172.21.15.28:0/3389780006 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f597806ac50 0x7f597806d110 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5988009bd0 tx=0x7f5988009240 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.105+0000 7f5994c00640 1 --2- 172.21.15.28:0/3389780006 >> [v2:172.21.15.191:6816/2503027971,v1:172.21.15.191:6817/2503027971] conn(0x7f5978072e70 0x7f5978075290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.106+0000 7fe257e00640 1 -- 172.21.15.28:0/1813373032 <== osd.4 v2:172.21.15.191:6800/2817102450 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fe240075940 con 0x7fe240072e70 2024-09-13T16:44:03.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.106+0000 7f5994c00640 1 -- 172.21.15.28:0/3389780006 --> [v2:172.21.15.191:6816/2503027971,v1:172.21.15.191:6817/2503027971] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5978075940 con 0x7f5978072e70 2024-09-13T16:44:03.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.106+0000 7f5994c00640 1 -- 172.21.15.28:0/3389780006 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f598408b740 con 0x7f5998157110 2024-09-13T16:44:03.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.106+0000 7f599d600640 1 --2- 172.21.15.28:0/3389780006 >> [v2:172.21.15.191:6816/2503027971,v1:172.21.15.191:6817/2503027971] conn(0x7f5978072e70 0x7f5978075290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.110+0000 7f599d600640 1 --2- 172.21.15.28:0/3389780006 >> [v2:172.21.15.191:6816/2503027971,v1:172.21.15.191:6817/2503027971] conn(0x7f5978072e70 0x7f5978075290 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.111+0000 7f5994c00640 1 -- 172.21.15.28:0/3389780006 <== osd.5 v2:172.21.15.191:6816/2503027971 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f5978075940 con 0x7f5978072e70 2024-09-13T16:44:03.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.112+0000 7f7890a00640 1 -- 172.21.15.28:0/2990707434 --> [v2:172.21.15.28:6816/886887303,v1:172.21.15.28:6817/886887303] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7864002cb0 con 0x7f786c072e00 2024-09-13T16:44:03.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.113+0000 7f7882a00640 1 -- 172.21.15.28:0/2990707434 <== osd.2 v2:172.21.15.28:6816/886887303 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7864002cb0 con 0x7f786c072e00 2024-09-13T16:44:03.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.115+0000 7f7877e00640 1 -- 172.21.15.28:0/2990707434 >> [v2:172.21.15.28:6816/886887303,v1:172.21.15.28:6817/886887303] conn(0x7f786c072e00 msgr2=0x7f786c075220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.115+0000 7f7877e00640 1 --2- 172.21.15.28:0/2990707434 >> [v2:172.21.15.28:6816/886887303,v1:172.21.15.28:6817/886887303] conn(0x7f786c072e00 0x7f786c075220 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.115+0000 7f7877e00640 1 -- 172.21.15.28:0/2990707434 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f786c06ac60 msgr2=0x7f786c06d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.115+0000 7f7877e00640 1 --2- 172.21.15.28:0/2990707434 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f786c06ac60 0x7f786c06d120 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f787c034000 tx=0x7f787c02e000 comp rx=0 tx=0).stop 2024-09-13T16:44:03.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.115+0000 7f7877e00640 1 -- 172.21.15.28:0/2990707434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78840e6610 msgr2=0x7f78840dbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.115+0000 7f7877e00640 1 --2- 172.21.15.28:0/2990707434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78840e6610 0x7f78840dbf10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f788c063830 tx=0x7f788c067a20 comp rx=0 tx=0).stop 2024-09-13T16:44:03.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.115+0000 7f788b200640 1 -- 172.21.15.28:0/2990707434 reap_dead start 2024-09-13T16:44:03.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.115+0000 7f7877e00640 1 -- 172.21.15.28:0/2990707434 shutdown_connections 2024-09-13T16:44:03.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.115+0000 7f7877e00640 1 -- 172.21.15.28:0/2990707434 >> 172.21.15.28:0/2990707434 conn(0x7f788404fc20 msgr2=0x7f7884006cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.116+0000 7f7877e00640 1 -- 172.21.15.28:0/2990707434 shutdown_connections 2024-09-13T16:44:03.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.116+0000 7f7877e00640 1 -- 172.21.15.28:0/2990707434 wait complete. 2024-09-13T16:44:03.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.119+0000 7fb23cc00640 1 -- 172.21.15.28:0/828383813 --> [v2:172.21.15.28:6808/23768570,v1:172.21.15.28:6809/23768570] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb210002cb0 con 0x7fb21807b6c0 2024-09-13T16:44:03.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.120+0000 7fb22f400640 1 -- 172.21.15.28:0/828383813 <== osd.1 v2:172.21.15.28:6808/23768570 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb210002cb0 con 0x7fb21807b6c0 2024-09-13T16:44:03.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.121+0000 7fb22cc00640 1 -- 172.21.15.28:0/828383813 >> [v2:172.21.15.28:6808/23768570,v1:172.21.15.28:6809/23768570] conn(0x7fb21807b6c0 msgr2=0x7fb21807dae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.121+0000 7fb22cc00640 1 --2- 172.21.15.28:0/828383813 >> [v2:172.21.15.28:6808/23768570,v1:172.21.15.28:6809/23768570] conn(0x7fb21807b6c0 0x7fb21807dae0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.122+0000 7fb22cc00640 1 -- 172.21.15.28:0/828383813 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb2180734a0 msgr2=0x7fb218075960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.122+0000 7fb22cc00640 1 --2- 172.21.15.28:0/828383813 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb2180734a0 0x7fb218075960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb228007fd0 tx=0x7fb228001d80 comp rx=0 tx=0).stop 2024-09-13T16:44:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.122+0000 7fb22cc00640 1 -- 172.21.15.28:0/828383813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb21c003990 msgr2=0x7fb23814eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.122+0000 7fb22cc00640 1 --2- 172.21.15.28:0/828383813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb21c003990 0x7fb23814eb80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb230032000 tx=0x7fb230004ab0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.122+0000 7fb237e00640 1 -- 172.21.15.28:0/828383813 reap_dead start 2024-09-13T16:44:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.122+0000 7fb22cc00640 1 -- 172.21.15.28:0/828383813 shutdown_connections 2024-09-13T16:44:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.122+0000 7fb22cc00640 1 -- 172.21.15.28:0/828383813 >> 172.21.15.28:0/828383813 conn(0x7fb23806c880 msgr2=0x7fb2380cd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.123+0000 7f89c7600640 1 -- 172.21.15.28:0/1085338538 --> [v2:172.21.15.28:6800/2964323435,v1:172.21.15.28:6801/2964323435] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f89b80e24c0 con 0x7f89a0072e80 2024-09-13T16:44:03.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.123+0000 7fb22cc00640 1 -- 172.21.15.28:0/828383813 shutdown_connections 2024-09-13T16:44:03.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.123+0000 7fb22cc00640 1 -- 172.21.15.28:0/828383813 wait complete. 2024-09-13T16:44:03.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.124+0000 7f89bcc00640 1 -- 172.21.15.28:0/1085338538 <== osd.0 v2:172.21.15.28:6800/2964323435 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f89c006ae90 con 0x7f89a0072e80 2024-09-13T16:44:03.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.124+0000 7f89c7600640 1 -- 172.21.15.28:0/1085338538 >> [v2:172.21.15.28:6800/2964323435,v1:172.21.15.28:6801/2964323435] conn(0x7f89a0072e80 msgr2=0x7f89a00752a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.124+0000 7f89c7600640 1 --2- 172.21.15.28:0/1085338538 >> [v2:172.21.15.28:6800/2964323435,v1:172.21.15.28:6801/2964323435] conn(0x7f89a0072e80 0x7f89a00752a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.126+0000 7f89c7600640 1 -- 172.21.15.28:0/1085338538 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f89a006ac60 msgr2=0x7f89a006d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.126+0000 7f89c7600640 1 --2- 172.21.15.28:0/1085338538 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f89a006ac60 0x7f89a006d120 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f89b4034000 tx=0x7f89b402e000 comp rx=0 tx=0).stop 2024-09-13T16:44:03.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.126+0000 7f89c7600640 1 -- 172.21.15.28:0/1085338538 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89b80d8830 msgr2=0x7f89b80dc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.126+0000 7f89c7600640 1 --2- 172.21.15.28:0/1085338538 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89b80d8830 0x7f89b80dc250 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00c500 tx=0x7f89ac00c9c0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.126+0000 7f89c5600640 1 -- 172.21.15.28:0/1085338538 reap_dead start 2024-09-13T16:44:03.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.127+0000 7f09e2a00640 1 -- 172.21.15.28:0/406124455 --> [v2:172.21.15.191:6808/3006595088,v1:172.21.15.191:6809/3006595088] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f09b4005c20 con 0x7f09b40015e0 2024-09-13T16:44:03.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.127+0000 7f09d8c00640 1 -- 172.21.15.28:0/406124455 <== osd.6 v2:172.21.15.191:6808/3006595088 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f09b4005c20 con 0x7f09b40015e0 2024-09-13T16:44:03.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.128+0000 7fa413200640 1 -- 172.21.15.28:0/3473069070 --> [v2:172.21.15.28:6824/2616036674,v1:172.21.15.28:6825/2616036674] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa40c06b310 con 0x7fa3ec072df0 2024-09-13T16:44:03.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.128+0000 7f09e2a00640 1 -- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:6808/3006595088,v1:172.21.15.191:6809/3006595088] conn(0x7f09b40015e0 msgr2=0x7f09b4003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.128+0000 7f09e2a00640 1 --2- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:6808/3006595088,v1:172.21.15.191:6809/3006595088] conn(0x7f09b40015e0 0x7f09b4003aa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.129+0000 7f89c7600640 1 -- 172.21.15.28:0/1085338538 shutdown_connections 2024-09-13T16:44:03.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.129+0000 7f89c7600640 1 -- 172.21.15.28:0/1085338538 >> 172.21.15.28:0/1085338538 conn(0x7f89b8054410 msgr2=0x7f89b80547f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.129+0000 7f09e2a00640 1 -- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f09b806ac30 msgr2=0x7f09b806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.129+0000 7f89c7600640 1 -- 172.21.15.28:0/1085338538 shutdown_connections 2024-09-13T16:44:03.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.129+0000 7f89c7600640 1 -- 172.21.15.28:0/1085338538 wait complete. 2024-09-13T16:44:03.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.129+0000 7f09e2a00640 1 --2- 172.21.15.28:0/406124455 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f09b806ac30 0x7f09b806d0f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f09c4034000 tx=0x7f09c402e000 comp rx=0 tx=0).stop 2024-09-13T16:44:03.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.129+0000 7f09e2a00640 1 -- 172.21.15.28:0/406124455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09dc0c65f0 msgr2=0x7f09dc071040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.129+0000 7f09e2a00640 1 --2- 172.21.15.28:0/406124455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09dc0c65f0 0x7f09dc071040 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00c4a0 tx=0x7f09cc00c960 comp rx=0 tx=0).stop 2024-09-13T16:44:03.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.129+0000 7f09e0a00640 1 -- 172.21.15.28:0/406124455 reap_dead start 2024-09-13T16:44:03.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.129+0000 7f09e2a00640 1 -- 172.21.15.28:0/406124455 shutdown_connections 2024-09-13T16:44:03.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.130+0000 7fa403e00640 1 -- 172.21.15.28:0/3473069070 <== osd.3 v2:172.21.15.28:6824/2616036674 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa3ec0758c0 con 0x7fa3ec072df0 2024-09-13T16:44:03.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.130+0000 7f09e2a00640 1 -- 172.21.15.28:0/406124455 >> 172.21.15.28:0/406124455 conn(0x7f09dc0ca6b0 msgr2=0x7f09dc148cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.132+0000 7f4e63200640 1 -- 172.21.15.28:0/3335021308 --> [v2:172.21.15.191:6824/1477327720,v1:172.21.15.191:6825/1477327720] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4e3c002cb0 con 0x7f4e44072e00 2024-09-13T16:44:03.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.132+0000 7f09e2a00640 1 -- 172.21.15.28:0/406124455 shutdown_connections 2024-09-13T16:44:03.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.132+0000 7f09e2a00640 1 -- 172.21.15.28:0/406124455 wait complete. 2024-09-13T16:44:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.133+0000 7fa413200640 1 -- 172.21.15.28:0/3473069070 >> [v2:172.21.15.28:6824/2616036674,v1:172.21.15.28:6825/2616036674] conn(0x7fa3ec072df0 msgr2=0x7fa3ec075210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.133+0000 7f4e5a000640 1 -- 172.21.15.28:0/3335021308 <== osd.7 v2:172.21.15.191:6824/1477327720 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4e3c002cb0 con 0x7f4e44072e00 2024-09-13T16:44:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.133+0000 7fa413200640 1 --2- 172.21.15.28:0/3473069070 >> [v2:172.21.15.28:6824/2616036674,v1:172.21.15.28:6825/2616036674] conn(0x7fa3ec072df0 0x7fa3ec075210 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.133+0000 7fa413200640 1 -- 172.21.15.28:0/3473069070 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa3ec06ac50 msgr2=0x7fa3ec06d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.134+0000 7fa413200640 1 --2- 172.21.15.28:0/3473069070 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa3ec06ac50 0x7fa3ec06d110 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0018b0 tx=0x7fa3fc009040 comp rx=0 tx=0).stop 2024-09-13T16:44:03.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.134+0000 7fa413200640 1 -- 172.21.15.28:0/3473069070 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa40c152f20 msgr2=0x7fa40c1e3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.134+0000 7fa413200640 1 --2- 172.21.15.28:0/3473069070 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa40c152f20 0x7fa40c1e3d80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800c410 tx=0x7fa3f800c8d0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.134+0000 7fa411200640 1 -- 172.21.15.28:0/3473069070 reap_dead start 2024-09-13T16:44:03.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.134+0000 7f4e63200640 1 -- 172.21.15.28:0/3335021308 >> [v2:172.21.15.191:6824/1477327720,v1:172.21.15.191:6825/1477327720] conn(0x7f4e44072e00 msgr2=0x7f4e44075220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.134+0000 7f4e63200640 1 --2- 172.21.15.28:0/3335021308 >> [v2:172.21.15.191:6824/1477327720,v1:172.21.15.191:6825/1477327720] conn(0x7f4e44072e00 0x7f4e44075220 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.136+0000 7fa413200640 1 -- 172.21.15.28:0/3473069070 shutdown_connections 2024-09-13T16:44:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.136+0000 7fa413200640 1 -- 172.21.15.28:0/3473069070 >> 172.21.15.28:0/3473069070 conn(0x7fa40c065c50 msgr2=0x7fa40c150740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.136+0000 7fa413200640 1 -- 172.21.15.28:0/3473069070 shutdown_connections 2024-09-13T16:44:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.137+0000 7fa413200640 1 -- 172.21.15.28:0/3473069070 wait complete. 2024-09-13T16:44:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.136+0000 7fe266a00640 1 -- 172.21.15.28:0/1813373032 --> [v2:172.21.15.191:6800/2817102450,v1:172.21.15.191:6801/2817102450] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe26015c140 con 0x7fe240072e70 2024-09-13T16:44:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.137+0000 7fe257e00640 1 -- 172.21.15.28:0/1813373032 <== osd.4 v2:172.21.15.191:6800/2817102450 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe26015c140 con 0x7fe240072e70 2024-09-13T16:44:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.138+0000 7fe266a00640 1 -- 172.21.15.28:0/1813373032 >> [v2:172.21.15.191:6800/2817102450,v1:172.21.15.191:6801/2817102450] conn(0x7fe240072e70 msgr2=0x7fe240075290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.138+0000 7fe266a00640 1 --2- 172.21.15.28:0/1813373032 >> [v2:172.21.15.191:6800/2817102450,v1:172.21.15.191:6801/2817102450] conn(0x7fe240072e70 0x7fe240075290 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.138+0000 7fe266a00640 1 -- 172.21.15.28:0/1813373032 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe24006acd0 msgr2=0x7fe24006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.138+0000 7fe266a00640 1 --2- 172.21.15.28:0/1813373032 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe24006acd0 0x7fe24006d190 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe250006fd0 tx=0x7fe250008040 comp rx=0 tx=0).stop 2024-09-13T16:44:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.138+0000 7fe266a00640 1 -- 172.21.15.28:0/1813373032 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2600c25c0 msgr2=0x7fe26020cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.138+0000 7fe266a00640 1 --2- 172.21.15.28:0/1813373032 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe2600c25c0 0x7fe26020cc00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe24c000c00 tx=0x7fe24c0046c0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.138+0000 7fe264a00640 1 -- 172.21.15.28:0/1813373032 reap_dead start 2024-09-13T16:44:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.139+0000 7fe266a00640 1 -- 172.21.15.28:0/1813373032 shutdown_connections 2024-09-13T16:44:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.139+0000 7fe266a00640 1 -- 172.21.15.28:0/1813373032 >> 172.21.15.28:0/1813373032 conn(0x7fe2600ca6b0 msgr2=0x7fe2600c8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.139+0000 7fe266a00640 1 -- 172.21.15.28:0/1813373032 shutdown_connections 2024-09-13T16:44:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.139+0000 7fe266a00640 1 -- 172.21.15.28:0/1813373032 wait complete. 2024-09-13T16:44:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.135+0000 7f4e63200640 1 -- 172.21.15.28:0/3335021308 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4e4406ac60 msgr2=0x7f4e4406d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.135+0000 7f4e63200640 1 --2- 172.21.15.28:0/3335021308 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4e4406ac60 0x7f4e4406d120 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4e54009c30 tx=0x7f4e54009190 comp rx=0 tx=0).stop 2024-09-13T16:44:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.139+0000 7f4e63200640 1 -- 172.21.15.28:0/3335021308 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e64160ee0 msgr2=0x7f4e641e6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.139+0000 7f4e63200640 1 --2- 172.21.15.28:0/3335021308 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e64160ee0 0x7f4e641e6240 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4e5000c3c0 tx=0x7f4e5000c880 comp rx=0 tx=0).stop 2024-09-13T16:44:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.139+0000 7f4e62800640 1 -- 172.21.15.28:0/3335021308 reap_dead start 2024-09-13T16:44:03.167 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-13T16:44:03.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T16:44:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.140+0000 7f4e63200640 1 -- 172.21.15.28:0/3335021308 shutdown_connections 2024-09-13T16:44:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.140+0000 7f4e63200640 1 -- 172.21.15.28:0/3335021308 >> 172.21.15.28:0/3335021308 conn(0x7f4e64069e80 msgr2=0x7f4e64156170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.140+0000 7f4e63200640 1 -- 172.21.15.28:0/3335021308 shutdown_connections 2024-09-13T16:44:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.140+0000 7f4e63200640 1 -- 172.21.15.28:0/3335021308 wait complete. 2024-09-13T16:44:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.142+0000 7f599f600640 1 -- 172.21.15.28:0/3389780006 --> [v2:172.21.15.191:6816/2503027971,v1:172.21.15.191:6817/2503027971] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f599815d380 con 0x7f5978072e70 2024-09-13T16:44:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.143+0000 7f5994c00640 1 -- 172.21.15.28:0/3389780006 <== osd.5 v2:172.21.15.191:6816/2503027971 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f59900075c0 con 0x7f5978072e70 2024-09-13T16:44:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.144+0000 7f599f600640 1 -- 172.21.15.28:0/3389780006 >> [v2:172.21.15.191:6816/2503027971,v1:172.21.15.191:6817/2503027971] conn(0x7f5978072e70 msgr2=0x7f5978075290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.144+0000 7f599f600640 1 --2- 172.21.15.28:0/3389780006 >> [v2:172.21.15.191:6816/2503027971,v1:172.21.15.191:6817/2503027971] conn(0x7f5978072e70 0x7f5978075290 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.145+0000 7f599f600640 1 -- 172.21.15.28:0/3389780006 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f597806ac50 msgr2=0x7f597806d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.145+0000 7f599f600640 1 --2- 172.21.15.28:0/3389780006 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f597806ac50 0x7f597806d110 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5988009bd0 tx=0x7f5988009240 comp rx=0 tx=0).stop 2024-09-13T16:44:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.145+0000 7f599f600640 1 -- 172.21.15.28:0/3389780006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5998157110 msgr2=0x7f59981f2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.145+0000 7f599f600640 1 --2- 172.21.15.28:0/3389780006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5998157110 0x7f59981f2f10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f598400c430 tx=0x7f598400c8f0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.145+0000 7f599d600640 1 -- 172.21.15.28:0/3389780006 reap_dead start 2024-09-13T16:44:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.145+0000 7f599f600640 1 -- 172.21.15.28:0/3389780006 shutdown_connections 2024-09-13T16:44:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.145+0000 7f599f600640 1 -- 172.21.15.28:0/3389780006 >> 172.21.15.28:0/3389780006 conn(0x7f59980ca6b0 msgr2=0x7f59980c8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.146+0000 7f599f600640 1 -- 172.21.15.28:0/3389780006 shutdown_connections 2024-09-13T16:44:03.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.146+0000 7f599f600640 1 -- 172.21.15.28:0/3389780006 wait complete. 2024-09-13T16:44:03.171 INFO:teuthology.orchestra.run.smithi028.stdout:55834574853 2024-09-13T16:44:03.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T16:44:03.172 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-13T16:44:03.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T16:44:03.173 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-13T16:44:03.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T16:44:03.175 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-13T16:44:03.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T16:44:03.175 INFO:teuthology.orchestra.run.smithi028.stdout:55834574853 2024-09-13T16:44:03.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T16:44:03.176 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-13T16:44:03.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T16:44:03.177 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-13T16:44:03.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T16:44:03.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:44:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.490+0000 7f400d200640 1 Processor -- start 2024-09-13T16:44:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.490+0000 7f400d200640 1 -- start start 2024-09-13T16:44:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.492+0000 7f400d200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40000503e0 0x7f40000e2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.493+0000 7f4005e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f40000d8970 0x7f4000006be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59536/0 (socket says 172.21.15.28:59536) 2024-09-13T16:44:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.493+0000 7f4005e00640 1 -- 172.21.15.28:0/1167962548 learned_addr learned my addr 172.21.15.28:0/1167962548 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.493+0000 7f400d200640 1 -- 172.21.15.28:0/1167962548 --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400004f280 con 0x7f40000d8970 2024-09-13T16:44:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.493+0000 7f400d200640 1 -- 172.21.15.28:0/1167962548 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400004ece0 con 0x7f40000072f0 2024-09-13T16:44:03.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.493+0000 7f400d200640 1 -- 172.21.15.28:0/1167962548 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4000039f10 con 0x7f40000503e0 2024-09-13T16:44:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.493+0000 7f4006800640 1 --2- 172.21.15.28:0/1167962548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40000503e0 0x7f40000e2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.494+0000 7f4004a00640 1 -- 172.21.15.28:0/1167962548 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200796219 0 0) 0x7f400004ece0 con 0x7f40000072f0 2024-09-13T16:44:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.494+0000 7f4004a00640 1 -- 172.21.15.28:0/1167962548 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fec003520 con 0x7f40000072f0 2024-09-13T16:44:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.494+0000 7f4006800640 1 -- 172.21.15.28:0/1167962548 >> v1:172.21.15.191:6789/0 conn(0x7f40000072f0 legacy=0x7f40000e1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.494+0000 7f4006800640 1 -- 172.21.15.28:0/1167962548 >> v1:172.21.15.28:6789/0 conn(0x7f40000d8970 legacy=0x7f4000006be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.494+0000 7f4004a00640 1 -- 172.21.15.28:0/1167962548 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247959729 0 0) 0x7f400004f280 con 0x7f40000d8970 2024-09-13T16:44:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.494+0000 7f4006800640 1 -- 172.21.15.28:0/1167962548 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400004efa0 con 0x7f40000503e0 2024-09-13T16:44:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.495+0000 7f2cde600640 1 Processor -- start 2024-09-13T16:44:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.496+0000 7f4006800640 1 --2- 172.21.15.28:0/1167962548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40000503e0 0x7f40000e2260 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3ff80097d0 tx=0x7f3ff803b240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9adb719b0e94573 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.495+0000 7f2cde600640 1 -- start start 2024-09-13T16:44:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.495+0000 7f3f4e800640 1 Processor -- start 2024-09-13T16:44:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.495+0000 7f3f4e800640 1 -- start start 2024-09-13T16:44:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.497+0000 7f4004a00640 1 -- 172.21.15.28:0/1167962548 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8043440 con 0x7f40000503e0 2024-09-13T16:44:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.497+0000 7f4004a00640 1 -- 172.21.15.28:0/1167962548 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff80435a0 con 0x7f40000503e0 2024-09-13T16:44:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.497+0000 7f4004a00640 1 -- 172.21.15.28:0/1167962548 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8043930 con 0x7f40000503e0 2024-09-13T16:44:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.497+0000 7f3f4e800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f400dccb0 0x7f3f4004f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.497+0000 7f3f4e800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3f40050600 0x7f3f4004fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.498+0000 7f400d200640 1 -- 172.21.15.28:0/1167962548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40000503e0 msgr2=0x7f40000e2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.498+0000 7f400d200640 1 --2- 172.21.15.28:0/1167962548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40000503e0 0x7f40000e2260 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3ff80097d0 tx=0x7f3ff803b240 comp rx=0 tx=0).stop 2024-09-13T16:44:03.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.498+0000 7f2cde600640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2cd0050740 0x7f2cd00da510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.498+0000 7f2cde600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2cd00059d0 0x7f2cd00d4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.498+0000 7f2cde600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cd0034220 0x7f2cd0034650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.498+0000 7f2cde600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd0007320 con 0x7f2cd00059d0 2024-09-13T16:44:03.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.498+0000 7f2cde600640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd00d4a70 con 0x7f2cd0050740 2024-09-13T16:44:03.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.498+0000 7f2cde600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd0032be0 con 0x7f2cd0034220 2024-09-13T16:44:03.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.498+0000 7f2cdd200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2cd0050740 0x7f2cd00da510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.498+0000 7f2cdd200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2cd0050740 0x7f2cd00da510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60688/0 (socket says 172.21.15.28:60688) 2024-09-13T16:44:03.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.498+0000 7f2cdd200640 1 -- 172.21.15.28:0/1152477278 learned_addr learned my addr 172.21.15.28:0/1152477278 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.498+0000 7f3f4e800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f400066d0 con 0x7f3f400500c0 2024-09-13T16:44:03.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.498+0000 7f3f4e800640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f40054e20 con 0x7f3f40050600 2024-09-13T16:44:03.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.498+0000 7f3f4e800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f400e6b20 con 0x7f3f400dccb0 2024-09-13T16:44:03.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.498+0000 7f3f4d400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f400dccb0 0x7f3f4004f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.498+0000 7f3f4d400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f400dccb0 0x7f3f4004f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37460/0 (socket says 172.21.15.28:37460) 2024-09-13T16:44:03.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.498+0000 7f3f4d400640 1 -- 172.21.15.28:0/4216912494 learned_addr learned my addr 172.21.15.28:0/4216912494 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.499+0000 7f2cd7e00640 1 --2- 172.21.15.28:0/1152477278 >> v2:172.21.15.28:3300/0 conn(0x7f2cd00059d0 0x7f2cd00d4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.499+0000 7f3f4ca00640 1 --2- 172.21.15.28:0/4216912494 >> v2:172.21.15.191:3300/0 conn(0x7f3f40050600 0x7f3f4004fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.499+0000 7f2cddc00640 1 --2- 172.21.15.28:0/1152477278 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cd0034220 0x7f2cd0034650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.499+0000 7f3f4ca00640 1 -- 172.21.15.28:0/4216912494 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f400dccb0 msgr2=0x7f3f4004f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.499+0000 7f3f4ca00640 1 --2- 172.21.15.28:0/4216912494 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f400dccb0 0x7f3f4004f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.499+0000 7f3f4ca00640 1 -- 172.21.15.28:0/4216912494 >> v1:172.21.15.28:6789/0 conn(0x7f3f400500c0 legacy=0x7f3f40006b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.499+0000 7f3f4ca00640 1 -- 172.21.15.28:0/4216912494 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f400e6e00 con 0x7f3f40050600 2024-09-13T16:44:03.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.499+0000 7f2cdd200640 1 -- 172.21.15.28:0/1152477278 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cd0034220 msgr2=0x7f2cd0034650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.499+0000 7f2cdd200640 1 --2- 172.21.15.28:0/1152477278 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cd0034220 0x7f2cd0034650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.499+0000 7f400d200640 1 -- 172.21.15.28:0/1167962548 shutdown_connections 2024-09-13T16:44:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.499+0000 7f400d200640 1 --2- 172.21.15.28:0/1167962548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40000503e0 0x7f40000e2260 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.499+0000 7f2cdd200640 1 -- 172.21.15.28:0/1152477278 >> v2:172.21.15.28:3300/0 conn(0x7f2cd00059d0 msgr2=0x7f2cd00d4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.499+0000 7f2cdd200640 1 --2- 172.21.15.28:0/1152477278 >> v2:172.21.15.28:3300/0 conn(0x7f2cd00059d0 0x7f2cd00d4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.499+0000 7f2cdd200640 1 -- 172.21.15.28:0/1152477278 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd0005fa0 con 0x7f2cd0050740 2024-09-13T16:44:03.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.499+0000 7f400d200640 1 -- 172.21.15.28:0/1167962548 >> 172.21.15.28:0/1167962548 conn(0x7f4000054040 msgr2=0x7f4000054450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.500+0000 7f400d200640 1 -- 172.21.15.28:0/1167962548 shutdown_connections 2024-09-13T16:44:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.500+0000 7f2cd7e00640 1 --2- 172.21.15.28:0/1152477278 >> v2:172.21.15.28:3300/0 conn(0x7f2cd00059d0 0x7f2cd00d4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.500+0000 7f400d200640 1 -- 172.21.15.28:0/1167962548 wait complete. 2024-09-13T16:44:03.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.501+0000 7f2cdd200640 1 --2- 172.21.15.28:0/1152477278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cd0050740 0x7f2cd00da510 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8074ff0 tx=0x7f2cd80a4a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5619598a5c272073 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.501+0000 7f3f4ca00640 1 --2- 172.21.15.28:0/4216912494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f40050600 0x7f3f4004fa60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c0097a0 tx=0x7f3f3c02a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=501024a0e76a5243 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.501+0000 7f3f47e00640 1 -- 172.21.15.28:0/4216912494 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c02ae40 con 0x7f3f40050600 2024-09-13T16:44:03.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.501+0000 7f2cd7400640 1 -- 172.21.15.28:0/1152477278 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd80a8900 con 0x7f2cd0050740 2024-09-13T16:44:03.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.502+0000 7f2cd7400640 1 -- 172.21.15.28:0/1152477278 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd806ae90 con 0x7f2cd0050740 2024-09-13T16:44:03.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.502+0000 7f2cd7400640 1 -- 172.21.15.28:0/1152477278 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd80a8ce0 con 0x7f2cd0050740 2024-09-13T16:44:03.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.502+0000 7f3f47e00640 1 -- 172.21.15.28:0/4216912494 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3c032440 con 0x7f3f40050600 2024-09-13T16:44:03.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.502+0000 7f3f47e00640 1 -- 172.21.15.28:0/4216912494 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c0327d0 con 0x7f3f40050600 2024-09-13T16:44:03.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.503+0000 7f2cde600640 1 -- 172.21.15.28:0/1152477278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cd0050740 msgr2=0x7f2cd00da510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.503+0000 7f2cde600640 1 --2- 172.21.15.28:0/1152477278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cd0050740 0x7f2cd00da510 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8074ff0 tx=0x7f2cd80a4a80 comp rx=0 tx=0).stop 2024-09-13T16:44:03.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.503+0000 7f2cde600640 1 -- 172.21.15.28:0/1152477278 shutdown_connections 2024-09-13T16:44:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.503+0000 7f3f4e800640 1 -- 172.21.15.28:0/4216912494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f40050600 msgr2=0x7f3f4004fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.503+0000 7f3f4e800640 1 --2- 172.21.15.28:0/4216912494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f40050600 0x7f3f4004fa60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c0097a0 tx=0x7f3f3c02a130 comp rx=0 tx=0).stop 2024-09-13T16:44:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.503+0000 7f2cde600640 1 --2- 172.21.15.28:0/1152477278 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cd0034220 0x7f2cd0034650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.503+0000 7f2cde600640 1 --2- 172.21.15.28:0/1152477278 >> v2:172.21.15.28:3300/0 conn(0x7f2cd00059d0 0x7f2cd00d4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.503+0000 7f2cde600640 1 --2- 172.21.15.28:0/1152477278 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cd0050740 0x7f2cd00da510 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.503+0000 7f4bcfe00640 1 Processor -- start 2024-09-13T16:44:03.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.503+0000 7f3f4e800640 1 -- 172.21.15.28:0/4216912494 shutdown_connections 2024-09-13T16:44:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.503+0000 7f3f4e800640 1 --2- 172.21.15.28:0/4216912494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f40050600 0x7f3f4004fa60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.503+0000 7f3f4e800640 1 --2- 172.21.15.28:0/4216912494 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f400dccb0 0x7f3f4004f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.503+0000 7f3f4e800640 1 -- 172.21.15.28:0/4216912494 >> 172.21.15.28:0/4216912494 conn(0x7f3f40054050 msgr2=0x7f3f40054460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.503+0000 7f2cde600640 1 -- 172.21.15.28:0/1152477278 >> 172.21.15.28:0/1152477278 conn(0x7f2cd004fc20 msgr2=0x7f2cd0050030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.504+0000 7f4bcfe00640 1 -- start start 2024-09-13T16:44:03.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.504+0000 7f3f4e800640 1 -- 172.21.15.28:0/4216912494 shutdown_connections 2024-09-13T16:44:03.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.504+0000 7f2cde600640 1 -- 172.21.15.28:0/1152477278 shutdown_connections 2024-09-13T16:44:03.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.504+0000 7f2cde600640 1 -- 172.21.15.28:0/1152477278 wait complete. 2024-09-13T16:44:03.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.504+0000 7f400d200640 1 Processor -- start 2024-09-13T16:44:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.504+0000 7f3f4e800640 1 -- 172.21.15.28:0/4216912494 wait complete. 2024-09-13T16:44:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.505+0000 7f400d200640 1 -- start start 2024-09-13T16:44:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.506+0000 7f4bcfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4bc80d8840 0x7f4bc8049920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.506+0000 7f4bcfe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc80502a0 0x7f4bc8049e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.506+0000 7f4bcfe00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc8050820 con 0x7f4bc8049190 2024-09-13T16:44:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.506+0000 7f4bcfe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc80e28b0 con 0x7f4bc80d8840 2024-09-13T16:44:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.506+0000 7f4bcfe00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80e1c30 con 0x7f4bc80502a0 2024-09-13T16:44:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.506+0000 7f4bcf400640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f4bc8049190 0x7f4bc80de700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:51528/0 (socket says 172.21.15.28:51528) 2024-09-13T16:44:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.507+0000 7f4bcf400640 1 -- 172.21.15.28:0/1997680343 learned_addr learned my addr 172.21.15.28:0/1997680343 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:03.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.507+0000 7f4bcea00640 1 --2- 172.21.15.28:0/1997680343 >> v2:172.21.15.28:3300/0 conn(0x7f4bc80d8840 0x7f4bc8049920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.507+0000 7f4bce000640 1 --2- 172.21.15.28:0/1997680343 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc80502a0 0x7f4bc8049e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.507+0000 7f3f4e800640 1 Processor -- start 2024-09-13T16:44:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.506+0000 7f2cde600640 1 Processor -- start 2024-09-13T16:44:03.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.506+0000 7f2cde600640 1 -- start start 2024-09-13T16:44:03.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.507+0000 7f4bce000640 1 -- 172.21.15.28:0/1997680343 >> v2:172.21.15.28:3300/0 conn(0x7f4bc80d8840 msgr2=0x7f4bc8049920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.507+0000 7f4bce000640 1 --2- 172.21.15.28:0/1997680343 >> v2:172.21.15.28:3300/0 conn(0x7f4bc80d8840 0x7f4bc8049920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.507+0000 7f3f4e800640 1 -- start start 2024-09-13T16:44:03.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.507+0000 7f4bce000640 1 -- 172.21.15.28:0/1997680343 >> v1:172.21.15.191:6789/0 conn(0x7f4bc8049190 legacy=0x7f4bc80de700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.507+0000 7eff81400640 1 Processor -- start 2024-09-13T16:44:03.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.507+0000 7f4bce000640 1 -- 172.21.15.28:0/1997680343 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc80e1f10 con 0x7f4bc80502a0 2024-09-13T16:44:03.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.508+0000 7eff81400640 1 -- start start 2024-09-13T16:44:03.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.508+0000 7f400d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40000072f0 0x7f400016f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.508+0000 7f400d200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40000503e0 0x7f40000dcd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.508+0000 7f2cde600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd00059d0 0x7f2cd00465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.508+0000 7f4005e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40000072f0 0x7f400016f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.508+0000 7f400d200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40000d8970 0x7f40000db7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.508+0000 7f2cde600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cd0034220 0x7f2cd0046f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f2cde600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cd0050740 0x7f2cd00bf770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.508+0000 7f400d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40000d2ad0 con 0x7f40000072f0 2024-09-13T16:44:03.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.508+0000 7f400d200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40000dd2b0 con 0x7f40000503e0 2024-09-13T16:44:03.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.508+0000 7f400d200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400004f510 con 0x7f40000d8970 2024-09-13T16:44:03.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.508+0000 7f4005e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40000072f0 0x7f400016f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41572/0 (socket says 172.21.15.28:41572) 2024-09-13T16:44:03.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f4005e00640 1 -- 172.21.15.28:0/4041695902 learned_addr learned my addr 172.21.15.28:0/4041695902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f2cdd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd00059d0 0x7f2cd00465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f4005400640 1 --2- 172.21.15.28:0/4041695902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40000503e0 0x7f40000dcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f4005e00640 1 -- 172.21.15.28:0/4041695902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40000d8970 msgr2=0x7f40000db7c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-13T16:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f4005e00640 1 --2- 172.21.15.28:0/4041695902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40000d8970 0x7f40000db7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f4005e00640 1 -- 172.21.15.28:0/4041695902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40000503e0 msgr2=0x7f40000dcd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f4005e00640 1 --2- 172.21.15.28:0/4041695902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40000503e0 0x7f40000dcd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f4005e00640 1 -- 172.21.15.28:0/4041695902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff8009400 con 0x7f40000072f0 2024-09-13T16:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f2cdd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd00059d0 0x7f2cd00465d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41576/0 (socket says 172.21.15.28:41576) 2024-09-13T16:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f2cd7e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cd0034220 0x7f2cd0046f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f2cd7e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cd0034220 0x7f2cd0046f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37480/0 (socket says 172.21.15.28:37480) 2024-09-13T16:44:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f3f4e800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f400500c0 0x7f3f4016f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f3f4e800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f40050600 0x7f3f400d8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f3f4e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f400dccb0 0x7f3f4004dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f3f4e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4016bd50 con 0x7f3f400dccb0 2024-09-13T16:44:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f3f4e800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4016fd50 con 0x7f3f400500c0 2024-09-13T16:44:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f4005400640 1 --2- 172.21.15.28:0/4041695902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40000503e0 0x7f40000dcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:44:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f3f4e800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4016d710 con 0x7f3f40050600 2024-09-13T16:44:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7f4005e00640 1 --2- 172.21.15.28:0/4041695902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40000072f0 0x7f400016f7f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f400804f480 tx=0x7f4008067a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7f2cde600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd00be520 con 0x7f2cd00059d0 2024-09-13T16:44:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7f2cdd200640 1 -- 172.21.15.28:0/1356275622 learned_addr learned my addr 172.21.15.28:0/1356275622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7f2cde600640 1 -- 172.21.15.28:0/1356275622 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd00be800 con 0x7f2cd0050740 2024-09-13T16:44:03.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.509+0000 7f4bce000640 1 --2- 172.21.15.28:0/1997680343 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc80502a0 0x7f4bc8049e60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0097a0 tx=0x7f4bbc02a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4fa9a8f8d8be967 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7f3f4d400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f400500c0 0x7f3f4016f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7f2cde600640 1 -- 172.21.15.28:0/1356275622 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd00beae0 con 0x7f2cd0034220 2024-09-13T16:44:03.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7f2cddc00640 1 --2- 172.21.15.28:0/1356275622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cd0050740 0x7f2cd00bf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7f3f4d400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f400500c0 0x7f3f4016f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60720/0 (socket says 172.21.15.28:60720) 2024-09-13T16:44:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7f3f4d400640 1 -- 172.21.15.28:0/875945410 learned_addr learned my addr 172.21.15.28:0/875945410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7f3f4ca00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f40050600 0x7f3f400d8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7f3f4de00640 1 --2- 172.21.15.28:0/875945410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f400dccb0 0x7f3f4004dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7eff81400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7eff7c06bfc0 0x7eff7c0c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7f3f4d400640 1 -- 172.21.15.28:0/875945410 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f40050600 msgr2=0x7f3f400d8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7eff81400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7c070370 con 0x7eff7c06bbe0 2024-09-13T16:44:03.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7eff81400640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7c1609e0 con 0x7eff7c156b80 2024-09-13T16:44:03.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7eff81400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff7c160cc0 con 0x7eff7c06bfc0 2024-09-13T16:44:03.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7f3f4d400640 1 --2- 172.21.15.28:0/875945410 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f40050600 0x7f3f400d8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7f3f4d400640 1 -- 172.21.15.28:0/875945410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f400dccb0 msgr2=0x7f3f4004dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7f3f4d400640 1 --2- 172.21.15.28:0/875945410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f400dccb0 0x7f3f4004dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7f3f4d400640 1 -- 172.21.15.28:0/875945410 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c009400 con 0x7f3f400500c0 2024-09-13T16:44:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.511+0000 7f3f4de00640 1 --2- 172.21.15.28:0/875945410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f400dccb0 0x7f3f4004dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.511+0000 7eff7be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7eff7c06bbe0 0x7eff7c160210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59542/0 (socket says 172.21.15.28:59542) 2024-09-13T16:44:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.511+0000 7eff7be00640 1 -- 172.21.15.28:0/4191820627 learned_addr learned my addr 172.21.15.28:0/4191820627 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.510+0000 7f4bcd600640 1 -- 172.21.15.28:0/1997680343 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc02ae60 con 0x7f4bc80502a0 2024-09-13T16:44:03.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.511+0000 7f4bcd600640 1 -- 172.21.15.28:0/1997680343 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bbc032440 con 0x7f4bc80502a0 2024-09-13T16:44:03.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.511+0000 7f4bcd600640 1 -- 172.21.15.28:0/1997680343 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc02ae60 con 0x7f4bc80502a0 2024-09-13T16:44:03.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.511+0000 7eff7b400640 1 --2- 172.21.15.28:0/4191820627 >> v2:172.21.15.28:3300/0 conn(0x7eff7c06bfc0 0x7eff7c0c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.511+0000 7f3ffe000640 1 -- 172.21.15.28:0/4041695902 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400806f5b0 con 0x7f40000072f0 2024-09-13T16:44:03.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.511+0000 7f3ffe000640 1 -- 172.21.15.28:0/4041695902 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400806ae90 con 0x7f40000072f0 2024-09-13T16:44:03.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.511+0000 7f3f4d400640 1 --2- 172.21.15.28:0/875945410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f400500c0 0x7f3f4016f810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3f4802c9a0 tx=0x7f3f48067310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.511+0000 7f2cdd200640 1 -- 172.21.15.28:0/1356275622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cd0034220 msgr2=0x7f2cd0046f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.511+0000 7f3f45600640 1 -- 172.21.15.28:0/875945410 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4808f760 con 0x7f3f400500c0 2024-09-13T16:44:03.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.511+0000 7f3f45600640 1 -- 172.21.15.28:0/875945410 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4806ae90 con 0x7f3f400500c0 2024-09-13T16:44:03.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.511+0000 7f3ffe000640 1 -- 172.21.15.28:0/4041695902 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400806fa10 con 0x7f40000072f0 2024-09-13T16:44:03.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.512+0000 7f3f4e800640 1 -- 172.21.15.28:0/875945410 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4016bf80 con 0x7f3f400500c0 2024-09-13T16:44:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.512+0000 7f4bcfe00640 1 -- 172.21.15.28:0/1997680343 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc80502a0 msgr2=0x7f4bc8049e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.512+0000 7f4bcfe00640 1 --2- 172.21.15.28:0/1997680343 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc80502a0 0x7f4bc8049e60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0097a0 tx=0x7f4bbc02a110 comp rx=0 tx=0).stop 2024-09-13T16:44:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.512+0000 7f3f4e800640 1 -- 172.21.15.28:0/875945410 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4016c580 con 0x7f3f400500c0 2024-09-13T16:44:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.512+0000 7f3f45600640 1 -- 172.21.15.28:0/875945410 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4808fac0 con 0x7f3f400500c0 2024-09-13T16:44:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.512+0000 7f400d200640 1 -- 172.21.15.28:0/4041695902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40000dc8e0 con 0x7f40000072f0 2024-09-13T16:44:03.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.512+0000 7eff7b400640 1 -- 172.21.15.28:0/4191820627 >> v1:172.21.15.191:6789/0 conn(0x7eff7c156b80 legacy=0x7eff7c070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.512+0000 7f2cdd200640 1 --2- 172.21.15.28:0/1356275622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cd0034220 0x7f2cd0046f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.512+0000 7f2cdd200640 1 -- 172.21.15.28:0/1356275622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cd0050740 msgr2=0x7f2cd00bf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.512+0000 7f2cdd200640 1 --2- 172.21.15.28:0/1356275622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cd0050740 0x7f2cd00bf770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.512+0000 7f2cdd200640 1 -- 172.21.15.28:0/1356275622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd8075340 con 0x7f2cd00059d0 2024-09-13T16:44:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.512+0000 7eff7aa00640 1 -- 172.21.15.28:0/4191820627 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109811656 0 0) 0x7eff7c070370 con 0x7eff7c06bbe0 2024-09-13T16:44:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.512+0000 7f400d200640 1 -- 172.21.15.28:0/4041695902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f400004f760 con 0x7f40000072f0 2024-09-13T16:44:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.512+0000 7f2cd7e00640 1 --2- 172.21.15.28:0/1356275622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cd0034220 0x7f2cd0046f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.512+0000 7f2cddc00640 1 --2- 172.21.15.28:0/1356275622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cd0050740 0x7f2cd00bf770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.513+0000 7f2cdd200640 1 --2- 172.21.15.28:0/1356275622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd00059d0 0x7f2cd00465d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8074ff0 tx=0x7f2cd8075230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.513+0000 7f7d53800640 1 Processor -- start 2024-09-13T16:44:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.513+0000 7eff7b400640 1 -- 172.21.15.28:0/4191820627 >> v1:172.21.15.28:6789/0 conn(0x7eff7c06bbe0 legacy=0x7eff7c160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.513+0000 7eff7b400640 1 -- 172.21.15.28:0/4191820627 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7c160fa0 con 0x7eff7c06bfc0 2024-09-13T16:44:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.513+0000 7f2cd4c00640 1 -- 172.21.15.28:0/1356275622 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd80b4040 con 0x7f2cd00059d0 2024-09-13T16:44:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.514+0000 7f2cde600640 1 -- 172.21.15.28:0/1356275622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd0045dd0 con 0x7f2cd00059d0 2024-09-13T16:44:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.515+0000 7f400d200640 1 -- 172.21.15.28:0/4041695902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4000050140 con 0x7f40000072f0 2024-09-13T16:44:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.514+0000 7f2cde600640 1 -- 172.21.15.28:0/1356275622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd00bedc0 con 0x7f2cd00059d0 2024-09-13T16:44:03.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.513+0000 7f4bcfe00640 1 -- 172.21.15.28:0/1997680343 shutdown_connections 2024-09-13T16:44:03.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.513+0000 7f4bcfe00640 1 --2- 172.21.15.28:0/1997680343 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc80502a0 0x7f4bc8049e60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.513+0000 7f4bcfe00640 1 --2- 172.21.15.28:0/1997680343 >> v2:172.21.15.28:3300/0 conn(0x7f4bc80d8840 0x7f4bc8049920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.513+0000 7f4bcfe00640 1 -- 172.21.15.28:0/1997680343 >> 172.21.15.28:0/1997680343 conn(0x7f4bc8054420 msgr2=0x7f4bc8054830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.514+0000 7eff7b400640 1 --2- 172.21.15.28:0/4191820627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff7c06bfc0 0x7eff7c0c67e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7eff68009720 tx=0x7eff6802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2e4f55d9fc12e57 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.513+0000 7f7d53800640 1 -- start start 2024-09-13T16:44:03.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.515+0000 7f7d53800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d4c157110 0x7f7d4c0c5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.515+0000 7f7d53800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7d4c1574f0 0x7f7d4c0c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.515+0000 7f7d53800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7d4c06d880 0x7f7d4c06dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.515+0000 7f7d53800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4c071c60 con 0x7f7d4c1574f0 2024-09-13T16:44:03.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.515+0000 7f7d53800640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4c0c1ff0 con 0x7f7d4c06d880 2024-09-13T16:44:03.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.515+0000 7f7d53800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c15dee0 con 0x7f7d4c157110 2024-09-13T16:44:03.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.516+0000 7f2cde600640 1 -- 172.21.15.28:0/1356275622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd00462f0 con 0x7f2cd00059d0 2024-09-13T16:44:03.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.516+0000 7f2cd4c00640 1 -- 172.21.15.28:0/1356275622 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd80b0b80 con 0x7f2cd00059d0 2024-09-13T16:44:03.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.516+0000 7f4bcfe00640 1 -- 172.21.15.28:0/1997680343 shutdown_connections 2024-09-13T16:44:03.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.516+0000 7f2cd4c00640 1 -- 172.21.15.28:0/1356275622 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd80b4040 con 0x7f2cd00059d0 2024-09-13T16:44:03.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.516+0000 7eff7aa00640 1 -- 172.21.15.28:0/4191820627 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6802ade0 con 0x7eff7c06bfc0 2024-09-13T16:44:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.516+0000 7f4bcfe00640 1 -- 172.21.15.28:0/1997680343 wait complete. 2024-09-13T16:44:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.515+0000 7f3f45600640 1 -- 172.21.15.28:0/875945410 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3f4808fc20 con 0x7f3f400500c0 2024-09-13T16:44:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.515+0000 7f3f4e800640 1 -- 172.21.15.28:0/875945410 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f400d7d70 con 0x7f3f400500c0 2024-09-13T16:44:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.516+0000 7f3f45600640 1 --2- 172.21.15.28:0/875945410 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3f2806ad00 0x7f3f2806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.516+0000 7f3f45600640 1 -- 172.21.15.28:0/875945410 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f48103310 con 0x7f3f400500c0 2024-09-13T16:44:03.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.517+0000 7f3f4ca00640 1 --2- 172.21.15.28:0/875945410 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3f2806ad00 0x7f3f2806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.517+0000 7f7d50e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d4c157110 0x7f7d4c0c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.517+0000 7f7d51800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7d4c06d880 0x7f7d4c06dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.517+0000 7f7d50e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d4c157110 0x7f7d4c0c5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37520/0 (socket says 172.21.15.28:37520) 2024-09-13T16:44:03.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.517+0000 7f7d43e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7d4c1574f0 0x7f7d4c0c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.517+0000 7f7d50e00640 1 -- 172.21.15.28:0/441135542 learned_addr learned my addr 172.21.15.28:0/441135542 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.517+0000 7f7d51800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7d4c06d880 0x7f7d4c06dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60738/0 (socket says 172.21.15.28:60738) 2024-09-13T16:44:03.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.518+0000 7f3f4ca00640 1 --2- 172.21.15.28:0/875945410 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3f2806ad00 0x7f3f2806d1c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c034000 tx=0x7f3f3c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.518+0000 7f7d43e00640 1 -- 172.21.15.28:0/441135542 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d4c157110 msgr2=0x7f7d4c0c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.518+0000 7f7d43e00640 1 --2- 172.21.15.28:0/441135542 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d4c157110 0x7f7d4c0c5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.519+0000 7f2cd4c00640 1 -- 172.21.15.28:0/1356275622 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2cd80b7070 con 0x7f2cd00059d0 2024-09-13T16:44:03.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.519+0000 7eff7aa00640 1 -- 172.21.15.28:0/4191820627 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff68032440 con 0x7eff7c06bfc0 2024-09-13T16:44:03.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.519+0000 7eff7aa00640 1 -- 172.21.15.28:0/4191820627 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6802ade0 con 0x7eff7c06bfc0 2024-09-13T16:44:03.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.519+0000 7f7d43e00640 1 -- 172.21.15.28:0/441135542 >> v2:172.21.15.191:3300/0 conn(0x7f7d4c06d880 msgr2=0x7f7d4c06dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.519+0000 7f7d43e00640 1 --2- 172.21.15.28:0/441135542 >> v2:172.21.15.191:3300/0 conn(0x7f7d4c06d880 0x7f7d4c06dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.519+0000 7f7d43e00640 1 -- 172.21.15.28:0/441135542 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c06d3f0 con 0x7f7d4c1574f0 2024-09-13T16:44:03.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.519+0000 7f7d50e00640 1 --2- 172.21.15.28:0/441135542 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d4c157110 0x7f7d4c0c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:03.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.519+0000 7f7d51800640 1 --2- 172.21.15.28:0/441135542 >> v2:172.21.15.191:3300/0 conn(0x7f7d4c06d880 0x7f7d4c06dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:03.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.519+0000 7eff81400640 1 -- 172.21.15.28:0/4191820627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff7c06bfc0 msgr2=0x7eff7c0c67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.519+0000 7eff81400640 1 --2- 172.21.15.28:0/4191820627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff7c06bfc0 0x7eff7c0c67e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7eff68009720 tx=0x7eff6802a0d0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.520+0000 7f3ffe000640 1 -- 172.21.15.28:0/4041695902 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f400806fb70 con 0x7f40000072f0 2024-09-13T16:44:03.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.520+0000 7eff81400640 1 -- 172.21.15.28:0/4191820627 shutdown_connections 2024-09-13T16:44:03.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.520+0000 7eff81400640 1 --2- 172.21.15.28:0/4191820627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff7c06bfc0 0x7eff7c0c67e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.520+0000 7eff81400640 1 -- 172.21.15.28:0/4191820627 >> 172.21.15.28:0/4191820627 conn(0x7eff7c0ca2f0 msgr2=0x7eff7c0ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.520+0000 7f2cd4c00640 1 --2- 172.21.15.28:0/1356275622 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f2cb806ac60 0x7f2cb806d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.520+0000 7f7d43e00640 1 --2- 172.21.15.28:0/441135542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d4c1574f0 0x7f7d4c0c5ac0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c073ef0 tx=0x7f7d3402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c604512772dd433 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.520+0000 7f2cd4c00640 1 -- 172.21.15.28:0/1356275622 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2cd8124480 con 0x7f2cd00059d0 2024-09-13T16:44:03.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.521+0000 7f2cd7e00640 1 --2- 172.21.15.28:0/1356275622 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f2cb806ac60 0x7f2cb806d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.521+0000 7f3f45600640 1 -- 172.21.15.28:0/875945410 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3f400d7d70 con 0x7f3f400500c0 2024-09-13T16:44:03.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.520+0000 7eff81400640 1 -- 172.21.15.28:0/4191820627 shutdown_connections 2024-09-13T16:44:03.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.520+0000 7eff81400640 1 -- 172.21.15.28:0/4191820627 wait complete. 2024-09-13T16:44:03.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.521+0000 7f3ffe000640 1 --2- 172.21.15.28:0/4041695902 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3fe806ad00 0x7f3fe806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.522+0000 7f3ffe000640 1 -- 172.21.15.28:0/4041695902 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4008103590 con 0x7f40000072f0 2024-09-13T16:44:03.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.522+0000 7f4bcfe00640 1 Processor -- start 2024-09-13T16:44:03.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.523+0000 7f4bcfe00640 1 -- start start 2024-09-13T16:44:03.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.523+0000 7f4005400640 1 --2- 172.21.15.28:0/4041695902 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3fe806ad00 0x7f3fe806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.523+0000 7eff81400640 1 Processor -- start 2024-09-13T16:44:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.523+0000 7eff81400640 1 -- start start 2024-09-13T16:44:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.524+0000 7f4005400640 1 --2- 172.21.15.28:0/4041695902 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3fe806ad00 0x7f3fe806d1c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f40000dc0e0 tx=0x7f3ff4009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.524+0000 7f2cd7e00640 1 --2- 172.21.15.28:0/1356275622 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f2cb806ac60 0x7f2cb806d120 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0045c30 tx=0x7f2cc4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.524+0000 7f4bcfe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc8049190 0x7f4bc80d0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.524+0000 7f4bcfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bc80502a0 0x7f4bc81741e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.524+0000 7f4bcfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bc80d8840 0x7f4bc8174720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.524+0000 7f4bcfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc8037fe0 con 0x7f4bc80d8840 2024-09-13T16:44:03.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.524+0000 7f4bcfe00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc8037b50 con 0x7f4bc80502a0 2024-09-13T16:44:03.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.524+0000 7f4bcfe00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80cfea0 con 0x7f4bc8049190 2024-09-13T16:44:03.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.524+0000 7f4bcea00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc8049190 0x7f4bc80d0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.524+0000 7f4bcea00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc8049190 0x7f4bc80d0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37536/0 (socket says 172.21.15.28:37536) 2024-09-13T16:44:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.524+0000 7f4bcea00640 1 -- 172.21.15.28:0/3659288185 learned_addr learned my addr 172.21.15.28:0/3659288185 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.524+0000 7f1e66000640 1 Processor -- start 2024-09-13T16:44:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.524+0000 7eff81400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff7c06bbe0 0x7eff7c0c37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.525+0000 7f4bce000640 1 --2- 172.21.15.28:0/3659288185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bc80502a0 0x7f4bc81741e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.525+0000 7eff81400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff7c06bfc0 0x7eff7c0c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.525+0000 7f4bcf400640 1 --2- 172.21.15.28:0/3659288185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bc80d8840 0x7f4bc8174720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.525+0000 7eff81400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff7c156b80 0x7eff7c0c2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.525+0000 7eff7b400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff7c06bfc0 0x7eff7c0c69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.525+0000 7f4bce000640 1 -- 172.21.15.28:0/3659288185 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc8049190 msgr2=0x7f4bc80d0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.525+0000 7f4bce000640 1 --2- 172.21.15.28:0/3659288185 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc8049190 0x7f4bc80d0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.525+0000 7f4bce000640 1 -- 172.21.15.28:0/3659288185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bc80d8840 msgr2=0x7f4bc8174720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.525+0000 7f4bce000640 1 --2- 172.21.15.28:0/3659288185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bc80d8840 0x7f4bc8174720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.525+0000 7f4bce000640 1 -- 172.21.15.28:0/3659288185 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc009400 con 0x7f4bc80502a0 2024-09-13T16:44:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.525+0000 7eff7b400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff7c06bfc0 0x7eff7c0c69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37550/0 (socket says 172.21.15.28:37550) 2024-09-13T16:44:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.525+0000 7eff7b400640 1 -- 172.21.15.28:0/914183604 learned_addr learned my addr 172.21.15.28:0/914183604 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.525+0000 7eff81400640 1 -- 172.21.15.28:0/914183604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c159880 con 0x7eff7c156b80 2024-09-13T16:44:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.525+0000 7eff81400640 1 -- 172.21.15.28:0/914183604 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c157ab0 con 0x7eff7c06bbe0 2024-09-13T16:44:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.525+0000 7eff81400640 1 -- 172.21.15.28:0/914183604 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c0c3d00 con 0x7eff7c06bfc0 2024-09-13T16:44:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.526+0000 7eff80a00640 1 --2- 172.21.15.28:0/914183604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff7c156b80 0x7eff7c0c2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.526+0000 7f4bce000640 1 --2- 172.21.15.28:0/3659288185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bc80502a0 0x7f4bc81741e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc031000 tx=0x7f4bbc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.526+0000 7eff7be00640 1 --2- 172.21.15.28:0/914183604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff7c06bbe0 0x7eff7c0c37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.526+0000 7f4bc6a00640 1 -- 172.21.15.28:0/3659288185 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc0327b0 con 0x7f4bc80502a0 2024-09-13T16:44:03.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.526+0000 7f4bc6a00640 1 -- 172.21.15.28:0/3659288185 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bbc032910 con 0x7f4bc80502a0 2024-09-13T16:44:03.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.526+0000 7f4bc6a00640 1 -- 172.21.15.28:0/3659288185 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc032c80 con 0x7f4bc80502a0 2024-09-13T16:44:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.525+0000 7f1e66000640 1 -- start start 2024-09-13T16:44:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.527+0000 7f3ffe000640 1 -- 172.21.15.28:0/4041695902 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f40080ce7e0 con 0x7f40000072f0 2024-09-13T16:44:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.527+0000 7f1e66000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e60152f50 0x7f1e60159690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.527+0000 7f1e66000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1e600c3660 0x7f1e600c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.527+0000 7f1e66000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e60159bd0 con 0x7f1e600c9b00 2024-09-13T16:44:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.527+0000 7f1e66000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e600c3f60 con 0x7f1e600c3660 2024-09-13T16:44:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.527+0000 7f1e66000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6015d0d0 con 0x7f1e60152f50 2024-09-13T16:44:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.527+0000 7f1e5ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1e600c3660 0x7f1e600c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.527+0000 7f1e5ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1e600c3660 0x7f1e600c95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41632/0 (socket says 172.21.15.28:41632) 2024-09-13T16:44:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.527+0000 7f1e5ea00640 1 -- 172.21.15.28:0/3698851790 learned_addr learned my addr 172.21.15.28:0/3698851790 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.527+0000 7eff80a00640 1 -- 172.21.15.28:0/914183604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff7c06bfc0 msgr2=0x7eff7c0c69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.527+0000 7eff80a00640 1 --2- 172.21.15.28:0/914183604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff7c06bfc0 0x7eff7c0c69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.527+0000 7eff80a00640 1 -- 172.21.15.28:0/914183604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff7c06bbe0 msgr2=0x7eff7c0c37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.527+0000 7eff80a00640 1 --2- 172.21.15.28:0/914183604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff7c06bbe0 0x7eff7c0c37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.527+0000 7eff80a00640 1 -- 172.21.15.28:0/914183604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff68009400 con 0x7eff7c156b80 2024-09-13T16:44:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.528+0000 7f4bcfe00640 1 -- 172.21.15.28:0/3659288185 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc80cd5b0 con 0x7f4bc80502a0 2024-09-13T16:44:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.528+0000 7f4bcfe00640 1 -- 172.21.15.28:0/3659288185 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc80cdb00 con 0x7f4bc80502a0 2024-09-13T16:44:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.528+0000 7f1e5f400640 1 --2- 172.21.15.28:0/3698851790 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e60152f50 0x7f1e60159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.528+0000 7f54c1600640 1 Processor -- start 2024-09-13T16:44:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.528+0000 7f54c1600640 1 -- start start 2024-09-13T16:44:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.528+0000 7f1e5ea00640 1 -- 172.21.15.28:0/3698851790 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e60152f50 msgr2=0x7f1e60159690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.528+0000 7f1e5ea00640 1 --2- 172.21.15.28:0/3698851790 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e60152f50 0x7f1e60159690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.528+0000 7f1e5ea00640 1 -- 172.21.15.28:0/3698851790 >> v1:172.21.15.28:6789/0 conn(0x7f1e600c9b00 legacy=0x7f1e600cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.528+0000 7f1e5ea00640 1 -- 172.21.15.28:0/3698851790 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6015c550 con 0x7f1e600c3660 2024-09-13T16:44:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.530+0000 7f4bcfe00640 1 -- 172.21.15.28:0/3659288185 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc8036df0 con 0x7f4bc80502a0 2024-09-13T16:44:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.530+0000 7eff80a00640 1 --2- 172.21.15.28:0/914183604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff7c156b80 0x7eff7c0c2140 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7eff7400c6d0 tx=0x7eff7400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.531+0000 7f54c1600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f54bc152f20 0x7f54bc0c34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.531+0000 7f54c1600640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f54bc0ca5d0 0x7f54bc0cb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.531+0000 7f54c1600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54bc0cb560 0x7f54bc15c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.531+0000 7f54c1600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54bc15d0e0 con 0x7f54bc152f20 2024-09-13T16:44:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.531+0000 7f54c1600640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54bc15ce00 con 0x7f54bc0ca5d0 2024-09-13T16:44:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.531+0000 7f54c1600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc06ad50 con 0x7f54bc0cb560 2024-09-13T16:44:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.531+0000 7f54baa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f54bc152f20 0x7f54bc0c34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.531+0000 7f54baa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f54bc152f20 0x7f54bc0c34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41642/0 (socket says 172.21.15.28:41642) 2024-09-13T16:44:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.531+0000 7f54baa00640 1 -- 172.21.15.28:0/3800892156 learned_addr learned my addr 172.21.15.28:0/3800892156 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.532+0000 7f54ba000640 1 --2- 172.21.15.28:0/3800892156 >> v2:172.21.15.191:3300/0 conn(0x7f54bc0ca5d0 0x7f54bc0cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.532+0000 7f54baa00640 1 -- 172.21.15.28:0/3800892156 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54bc0cb560 msgr2=0x7f54bc15c150 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-13T16:44:03.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.531+0000 7f1e5ea00640 1 --2- 172.21.15.28:0/3698851790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e600c3660 0x7f1e600c95c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009870 tx=0x7f1e4c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8156c9f14c6f02e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.532+0000 7f54baa00640 1 --2- 172.21.15.28:0/3800892156 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54bc0cb560 0x7f54bc15c150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.532+0000 7f54baa00640 1 -- 172.21.15.28:0/3800892156 >> v2:172.21.15.191:3300/0 conn(0x7f54bc0ca5d0 msgr2=0x7f54bc0cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:03.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.532+0000 7f54baa00640 1 --2- 172.21.15.28:0/3800892156 >> v2:172.21.15.191:3300/0 conn(0x7f54bc0ca5d0 0x7f54bc0cb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.532+0000 7f54baa00640 1 -- 172.21.15.28:0/3800892156 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54bc06b030 con 0x7f54bc152f20 2024-09-13T16:44:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.532+0000 7f4bc6a00640 1 -- 172.21.15.28:0/3659288185 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4bbc039040 con 0x7f4bc80502a0 2024-09-13T16:44:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.533+0000 7f4bc6a00640 1 --2- 172.21.15.28:0/3659288185 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4bb006acd0 0x7f4bb006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.533+0000 7f4bc6a00640 1 -- 172.21.15.28:0/3659288185 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4bbc0a9bd0 con 0x7f4bc80502a0 2024-09-13T16:44:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.534+0000 7f7d4be00640 1 -- 172.21.15.28:0/441135542 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34031600 con 0x7f7d4c1574f0 2024-09-13T16:44:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.534+0000 7eff73e00640 1 -- 172.21.15.28:0/914183604 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff74003b70 con 0x7eff7c156b80 2024-09-13T16:44:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.534+0000 7eff73e00640 1 -- 172.21.15.28:0/914183604 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff74003cd0 con 0x7eff7c156b80 2024-09-13T16:44:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.534+0000 7eff73e00640 1 -- 172.21.15.28:0/914183604 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff740052b0 con 0x7eff7c156b80 2024-09-13T16:44:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.534+0000 7f7d4be00640 1 -- 172.21.15.28:0/441135542 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3402d050 con 0x7f7d4c1574f0 2024-09-13T16:44:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.534+0000 7f7d4be00640 1 -- 172.21.15.28:0/441135542 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34031980 con 0x7f7d4c1574f0 2024-09-13T16:44:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.534+0000 7f54baa00640 1 --2- 172.21.15.28:0/3800892156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc152f20 0x7f54bc0c34b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f54bc156810 tx=0x7f54ac02a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ff5ec8cd5245342 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7eff81400640 1 -- 172.21.15.28:0/914183604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff7c157810 con 0x7eff7c156b80 2024-09-13T16:44:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7eff81400640 1 -- 172.21.15.28:0/914183604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff7c1f6140 con 0x7eff7c156b80 2024-09-13T16:44:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.534+0000 7f1e5e000640 1 -- 172.21.15.28:0/3698851790 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c032440 con 0x7f1e600c3660 2024-09-13T16:44:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.534+0000 7f1e5e000640 1 -- 172.21.15.28:0/3698851790 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e4c0325a0 con 0x7f1e600c3660 2024-09-13T16:44:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.534+0000 7f1e5e000640 1 -- 172.21.15.28:0/3698851790 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c032970 con 0x7f1e600c3660 2024-09-13T16:44:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7f7d53800640 1 -- 172.21.15.28:0/441135542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d4c1574f0 msgr2=0x7f7d4c0c5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7f7d53800640 1 --2- 172.21.15.28:0/441135542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d4c1574f0 0x7f7d4c0c5ac0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c073ef0 tx=0x7f7d3402a5d0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7f1e66000640 1 -- 172.21.15.28:0/3698851790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e600c3660 msgr2=0x7f1e600c95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7f1e66000640 1 --2- 172.21.15.28:0/3698851790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e600c3660 0x7f1e600c95c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009870 tx=0x7f1e4c02a230 comp rx=0 tx=0).stop 2024-09-13T16:44:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.534+0000 7f4bcea00640 1 --2- 172.21.15.28:0/3659288185 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4bb006acd0 0x7f4bb006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.534+0000 7f54b9600640 1 -- 172.21.15.28:0/3800892156 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac032440 con 0x7f54bc152f20 2024-09-13T16:44:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7f54b9600640 1 -- 172.21.15.28:0/3800892156 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ac0325a0 con 0x7f54bc152f20 2024-09-13T16:44:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7f54b9600640 1 -- 172.21.15.28:0/3800892156 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac032930 con 0x7f54bc152f20 2024-09-13T16:44:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7f2cd4c00640 1 -- 172.21.15.28:0/1356275622 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2cd00462f0 con 0x7f2cd00059d0 2024-09-13T16:44:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7f7d53800640 1 -- 172.21.15.28:0/441135542 shutdown_connections 2024-09-13T16:44:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7f7d53800640 1 --2- 172.21.15.28:0/441135542 >> v2:172.21.15.191:3300/0 conn(0x7f7d4c06d880 0x7f7d4c06dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.536+0000 7f54c1600640 1 -- 172.21.15.28:0/3800892156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc152f20 msgr2=0x7f54bc0c34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.536+0000 7f54c1600640 1 --2- 172.21.15.28:0/3800892156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc152f20 0x7f54bc0c34b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f54bc156810 tx=0x7f54ac02a200 comp rx=0 tx=0).stop 2024-09-13T16:44:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.536+0000 7f54c1600640 1 -- 172.21.15.28:0/3800892156 shutdown_connections 2024-09-13T16:44:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.536+0000 7f54c1600640 1 --2- 172.21.15.28:0/3800892156 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54bc0cb560 0x7f54bc15c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.536+0000 7f54c1600640 1 --2- 172.21.15.28:0/3800892156 >> v2:172.21.15.191:3300/0 conn(0x7f54bc0ca5d0 0x7f54bc0cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7f7d53800640 1 --2- 172.21.15.28:0/441135542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d4c1574f0 0x7f7d4c0c5ac0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7f7d53800640 1 --2- 172.21.15.28:0/441135542 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d4c157110 0x7f7d4c0c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7f7d53800640 1 -- 172.21.15.28:0/441135542 >> 172.21.15.28:0/441135542 conn(0x7f7d4c0ca6b0 msgr2=0x7f7d4c0caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.536+0000 7f7d53800640 1 -- 172.21.15.28:0/441135542 shutdown_connections 2024-09-13T16:44:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7f1e66000640 1 -- 172.21.15.28:0/3698851790 shutdown_connections 2024-09-13T16:44:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7f1e66000640 1 --2- 172.21.15.28:0/3698851790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e600c3660 0x7f1e600c95c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7f1e66000640 1 --2- 172.21.15.28:0/3698851790 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e60152f50 0x7f1e60159690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.535+0000 7f1e66000640 1 -- 172.21.15.28:0/3698851790 >> 172.21.15.28:0/3698851790 conn(0x7f1e60071790 msgr2=0x7f1e6006b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.536+0000 7f1e66000640 1 -- 172.21.15.28:0/3698851790 shutdown_connections 2024-09-13T16:44:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.536+0000 7f54c1600640 1 --2- 172.21.15.28:0/3800892156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc152f20 0x7f54bc0c34b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.536+0000 7f54c1600640 1 -- 172.21.15.28:0/3800892156 >> 172.21.15.28:0/3800892156 conn(0x7f54bc065c50 msgr2=0x7f54bc066060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.538+0000 7eff73e00640 1 -- 172.21.15.28:0/914183604 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7eff74005620 con 0x7eff7c156b80 2024-09-13T16:44:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.536+0000 7f4bcea00640 1 --2- 172.21.15.28:0/3659288185 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4bb006acd0 0x7f4bb006d190 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009c00 tx=0x7f4bc0009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.538+0000 7f4bc6a00640 1 -- 172.21.15.28:0/3659288185 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4bbc074e20 con 0x7f4bc80502a0 2024-09-13T16:44:03.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.536+0000 7f7d53800640 1 -- 172.21.15.28:0/441135542 wait complete. 2024-09-13T16:44:03.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.539+0000 7f7d53800640 1 Processor -- start 2024-09-13T16:44:03.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.536+0000 7f1e66000640 1 -- 172.21.15.28:0/3698851790 wait complete. 2024-09-13T16:44:03.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.539+0000 7f1e66000640 1 Processor -- start 2024-09-13T16:44:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.536+0000 7f54c1600640 1 -- 172.21.15.28:0/3800892156 shutdown_connections 2024-09-13T16:44:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.537+0000 7f54c1600640 1 -- 172.21.15.28:0/3800892156 wait complete. 2024-09-13T16:44:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.539+0000 7f54c1600640 1 Processor -- start 2024-09-13T16:44:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.539+0000 7f1e66000640 1 -- start start 2024-09-13T16:44:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.539+0000 7f7d53800640 1 -- start start 2024-09-13T16:44:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.539+0000 7eff81400640 1 -- 172.21.15.28:0/914183604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff7c0677a0 con 0x7eff7c156b80 2024-09-13T16:44:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.539+0000 7f54c1600640 1 -- start start 2024-09-13T16:44:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.541+0000 7f1e66000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e600c3660 0x7f1e600cca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.541+0000 7f7d53800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d4c06d880 0x7f7d4c158530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.541+0000 7eff73e00640 1 --2- 172.21.15.28:0/914183604 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7eff5c06ad00 0x7eff5c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.541+0000 7eff73e00640 1 -- 172.21.15.28:0/914183604 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff740892e0 con 0x7eff7c156b80 2024-09-13T16:44:03.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.541+0000 7f54c1600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54bc0ca5d0 0x7f54bc14ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f54c1600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54bc0cb560 0x7f54bc0b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f54baa00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54bc0ca5d0 0x7f54bc14ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f54baa00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54bc0ca5d0 0x7f54bc14ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37586/0 (socket says 172.21.15.28:37586) 2024-09-13T16:44:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f54baa00640 1 -- 172.21.15.28:0/301792768 learned_addr learned my addr 172.21.15.28:0/301792768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f54ba000640 1 --2- 172.21.15.28:0/301792768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54bc0cb560 0x7f54bc0b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f54c1600640 1 --2- 172.21.15.28:0/301792768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc152f20 0x7f54bc1fc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f7d53800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d4c157110 0x7f7d4c1595a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f7d53800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d4c1574f0 0x7f7d4c1f27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f7d51800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d4c1574f0 0x7f7d4c1f27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f7d51800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d4c1574f0 0x7f7d4c1f27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60806/0 (socket says 172.21.15.28:60806) 2024-09-13T16:44:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f7d51800640 1 -- 172.21.15.28:0/2314503442 learned_addr learned my addr 172.21.15.28:0/2314503442 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f7d43e00640 1 --2- 172.21.15.28:0/2314503442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d4c157110 0x7f7d4c1595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f7d53800640 1 -- 172.21.15.28:0/2314503442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c1f5d60 con 0x7f7d4c06d880 2024-09-13T16:44:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f7d53800640 1 -- 172.21.15.28:0/2314503442 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c1f68e0 con 0x7f7d4c1574f0 2024-09-13T16:44:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f7d53800640 1 -- 172.21.15.28:0/2314503442 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c161200 con 0x7f7d4c157110 2024-09-13T16:44:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f1e66000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e600c9b00 0x7f1e60148b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f1e5f400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e600c3660 0x7f1e600cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f1e66000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e60152f50 0x7f1e6015b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f1e5ea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e600c9b00 0x7f1e60148b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f1e5ea00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e600c9b00 0x7f1e60148b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60792/0 (socket says 172.21.15.28:60792) 2024-09-13T16:44:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f1e66000640 1 -- 172.21.15.28:0/3247433571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e600cdf60 con 0x7f1e60152f50 2024-09-13T16:44:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f1e66000640 1 -- 172.21.15.28:0/3247433571 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e600cc5c0 con 0x7f1e600c9b00 2024-09-13T16:44:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f1e66000640 1 -- 172.21.15.28:0/3247433571 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e600ccf90 con 0x7f1e600c3660 2024-09-13T16:44:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.542+0000 7f1e5ea00640 1 -- 172.21.15.28:0/3247433571 learned_addr learned my addr 172.21.15.28:0/3247433571 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.543+0000 7f1e5f400640 1 -- 172.21.15.28:0/3247433571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e600c9b00 msgr2=0x7f1e60148b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.543+0000 7f1e5f400640 1 --2- 172.21.15.28:0/3247433571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e600c9b00 0x7f1e60148b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.543+0000 7f1e5f400640 1 -- 172.21.15.28:0/3247433571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e60152f50 msgr2=0x7f1e6015b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.543+0000 7f1e5f400640 1 --2- 172.21.15.28:0/3247433571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e60152f50 0x7f1e6015b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.543+0000 7f1e5f400640 1 -- 172.21.15.28:0/3247433571 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e4c0094d0 con 0x7f1e600c3660 2024-09-13T16:44:03.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.543+0000 7f54c1600640 1 -- 172.21.15.28:0/301792768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc1fed40 con 0x7f54bc152f20 2024-09-13T16:44:03.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.543+0000 7f54c1600640 1 -- 172.21.15.28:0/301792768 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc1ff250 con 0x7f54bc0cb560 2024-09-13T16:44:03.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.543+0000 7f54c1600640 1 -- 172.21.15.28:0/301792768 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc0c8170 con 0x7f54bc0ca5d0 2024-09-13T16:44:03.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.544+0000 7f7d43e00640 1 -- 172.21.15.28:0/2314503442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d4c1574f0 msgr2=0x7f7d4c1f27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.544+0000 7f7d43e00640 1 --2- 172.21.15.28:0/2314503442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d4c1574f0 0x7f7d4c1f27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.544+0000 7eff7be00640 1 --2- 172.21.15.28:0/914183604 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7eff5c06ad00 0x7eff5c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.544+0000 7f7d43e00640 1 -- 172.21.15.28:0/2314503442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d4c06d880 msgr2=0x7f7d4c158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.544+0000 7f7d43e00640 1 --2- 172.21.15.28:0/2314503442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d4c06d880 0x7f7d4c158530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.544+0000 7f54ba000640 1 -- 172.21.15.28:0/301792768 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54bc0ca5d0 msgr2=0x7f54bc14ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.544+0000 7f54ba000640 1 --2- 172.21.15.28:0/301792768 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54bc0ca5d0 0x7f54bc14ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.544+0000 7f54ba000640 1 -- 172.21.15.28:0/301792768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc152f20 msgr2=0x7f54bc1fc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.544+0000 7f54ba000640 1 --2- 172.21.15.28:0/301792768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc152f20 0x7f54bc1fc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.544+0000 7f7d43e00640 1 -- 172.21.15.28:0/2314503442 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d44009400 con 0x7f7d4c157110 2024-09-13T16:44:03.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.544+0000 7f1e5f400640 1 --2- 172.21.15.28:0/3247433571 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e600c3660 0x7f1e600cca50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1e5000c5e0 tx=0x7f1e5000caa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.544+0000 7f54ba000640 1 -- 172.21.15.28:0/301792768 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac0094d0 con 0x7f54bc0cb560 2024-09-13T16:44:03.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.545+0000 7f1e57400640 1 -- 172.21.15.28:0/3247433571 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e50004220 con 0x7f1e600c3660 2024-09-13T16:44:03.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.545+0000 7f1e66000640 1 -- 172.21.15.28:0/3247433571 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e6015b700 con 0x7f1e600c3660 2024-09-13T16:44:03.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.545+0000 7f1e66000640 1 -- 172.21.15.28:0/3247433571 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e6015b140 con 0x7f1e600c3660 2024-09-13T16:44:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.545+0000 7f54ba000640 1 --2- 172.21.15.28:0/301792768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54bc0cb560 0x7f54bc0b3ef0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f54a800d5f0 tx=0x7f54a800dab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.545+0000 7f1e57400640 1 -- 172.21.15.28:0/3247433571 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e50004380 con 0x7f1e600c3660 2024-09-13T16:44:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.545+0000 7f1e57400640 1 -- 172.21.15.28:0/3247433571 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e50003850 con 0x7f1e600c3660 2024-09-13T16:44:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.546+0000 7f7d43e00640 1 --2- 172.21.15.28:0/2314503442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d4c157110 0x7f7d4c1595a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d34031000 tx=0x7f7d34004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.546+0000 7eff7be00640 1 --2- 172.21.15.28:0/914183604 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7eff5c06ad00 0x7eff5c06d1c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7eff6c000f60 tx=0x7eff6c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.547+0000 7f1e66000640 1 -- 172.21.15.28:0/3247433571 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e38005250 con 0x7f1e600c3660 2024-09-13T16:44:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.546+0000 7f54b2a00640 1 -- 172.21.15.28:0/301792768 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a80125f0 con 0x7f54bc0cb560 2024-09-13T16:44:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.547+0000 7f7d49600640 1 -- 172.21.15.28:0/2314503442 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3403c830 con 0x7f7d4c157110 2024-09-13T16:44:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.547+0000 7f7d49600640 1 -- 172.21.15.28:0/2314503442 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3403c9b0 con 0x7f7d4c157110 2024-09-13T16:44:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.547+0000 7f54b2a00640 1 -- 172.21.15.28:0/301792768 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a8012750 con 0x7f54bc0cb560 2024-09-13T16:44:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.547+0000 7f54c1600640 1 -- 172.21.15.28:0/301792768 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54bc1fef70 con 0x7f54bc0cb560 2024-09-13T16:44:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.547+0000 7f54c1600640 1 -- 172.21.15.28:0/301792768 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54bc0c7df0 con 0x7f54bc0cb560 2024-09-13T16:44:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.547+0000 7f7d53800640 1 -- 172.21.15.28:0/2314503442 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3c0094d0 con 0x7f7d4c157110 2024-09-13T16:44:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.548+0000 7f54b2a00640 1 -- 172.21.15.28:0/301792768 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a8012b00 con 0x7f54bc0cb560 2024-09-13T16:44:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.548+0000 7f7d53800640 1 -- 172.21.15.28:0/2314503442 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d34009a70 con 0x7f7d4c157110 2024-09-13T16:44:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.548+0000 7f7d49600640 1 -- 172.21.15.28:0/2314503442 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3403cd60 con 0x7f7d4c157110 2024-09-13T16:44:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.549+0000 7f1e57400640 1 -- 172.21.15.28:0/3247433571 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1e500039b0 con 0x7f1e600c3660 2024-09-13T16:44:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.551+0000 7f1e57400640 1 --2- 172.21.15.28:0/3247433571 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1e4006ac00 0x7f1e4006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.551+0000 7f1e57400640 1 -- 172.21.15.28:0/3247433571 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e500893c0 con 0x7f1e600c3660 2024-09-13T16:44:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.551+0000 7f7d53800640 1 -- 172.21.15.28:0/2314503442 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d24005250 con 0x7f7d4c157110 2024-09-13T16:44:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.552+0000 7f7d49600640 1 -- 172.21.15.28:0/2314503442 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7d3404e3f0 con 0x7f7d4c157110 2024-09-13T16:44:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.552+0000 7eff73e00640 1 -- 172.21.15.28:0/914183604 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7eff74054530 con 0x7eff7c156b80 2024-09-13T16:44:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.550+0000 7f54c1600640 1 -- 172.21.15.28:0/301792768 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5494005250 con 0x7f54bc0cb560 2024-09-13T16:44:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.552+0000 7f1e5ea00640 1 --2- 172.21.15.28:0/3247433571 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1e4006ac00 0x7f1e4006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.553+0000 7f1e5ea00640 1 --2- 172.21.15.28:0/3247433571 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1e4006ac00 0x7f1e4006d0c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1e6014db70 tx=0x7f1e4c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.553+0000 7f7d49600640 1 --2- 172.21.15.28:0/2314503442 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7d2806ac30 0x7f7d2806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.553+0000 7f7d49600640 1 -- 172.21.15.28:0/2314503442 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d340aace0 con 0x7f7d4c157110 2024-09-13T16:44:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.553+0000 7f54b2a00640 1 -- 172.21.15.28:0/301792768 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f54a8012c60 con 0x7f54bc0cb560 2024-09-13T16:44:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.554+0000 7f54b2a00640 1 --2- 172.21.15.28:0/301792768 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f549c06ac50 0x7f549c06d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.554+0000 7f54b2a00640 1 -- 172.21.15.28:0/301792768 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54a8089600 con 0x7f54bc0cb560 2024-09-13T16:44:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.554+0000 7f7d50e00640 1 --2- 172.21.15.28:0/2314503442 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7d2806ac30 0x7f7d2806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.554+0000 7f54baa00640 1 --2- 172.21.15.28:0/301792768 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f549c06ac50 0x7f549c06d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.555+0000 7f7d50e00640 1 --2- 172.21.15.28:0/2314503442 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7d2806ac30 0x7f7d2806d0f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c006f40 tx=0x7f7d3c006e90 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.555+0000 7f1e57400640 1 -- 172.21.15.28:0/3247433571 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1e500545b0 con 0x7f1e600c3660 2024-09-13T16:44:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.556+0000 7f54baa00640 1 --2- 172.21.15.28:0/301792768 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f549c06ac50 0x7f549c06d110 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f54ac02a240 tx=0x7f54ac02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.558+0000 7f54b2a00640 1 -- 172.21.15.28:0/301792768 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f54a80548c0 con 0x7f54bc0cb560 2024-09-13T16:44:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.559+0000 7f7d49600640 1 -- 172.21.15.28:0/2314503442 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7d340b1080 con 0x7f7d4c157110 2024-09-13T16:44:03.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.939+0000 7f400d200640 1 -- 172.21.15.28:0/4041695902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f4000052e20 con 0x7f40000072f0 2024-09-13T16:44:03.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.940+0000 7f3ffe000640 1 -- 172.21.15.28:0/4041695902 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4008099090 con 0x7f40000072f0 2024-09-13T16:44:03.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.940+0000 7f3f4e800640 1 -- 172.21.15.28:0/875945410 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f3f400d7d70 con 0x7f3f400500c0 2024-09-13T16:44:03.942 INFO:teuthology.orchestra.run.smithi028.stdout:55834574851 2024-09-13T16:44:03.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.941+0000 7f3f45600640 1 -- 172.21.15.28:0/875945410 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3f48097020 con 0x7f3f400500c0 2024-09-13T16:44:03.945 INFO:teuthology.orchestra.run.smithi028.stdout:55834574851 2024-09-13T16:44:03.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.944+0000 7f400d200640 1 -- 172.21.15.28:0/4041695902 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3fe806ad00 msgr2=0x7f3fe806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.944+0000 7f400d200640 1 --2- 172.21.15.28:0/4041695902 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3fe806ad00 0x7f3fe806d1c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f40000dc0e0 tx=0x7f3ff4009040 comp rx=0 tx=0).stop 2024-09-13T16:44:03.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.944+0000 7f400d200640 1 -- 172.21.15.28:0/4041695902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40000072f0 msgr2=0x7f400016f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.944+0000 7f400d200640 1 --2- 172.21.15.28:0/4041695902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40000072f0 0x7f400016f7f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f400804f480 tx=0x7f4008067a20 comp rx=0 tx=0).stop 2024-09-13T16:44:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.944+0000 7f3f4e800640 1 -- 172.21.15.28:0/875945410 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3f2806ad00 msgr2=0x7f3f2806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.945+0000 7f3f4e800640 1 --2- 172.21.15.28:0/875945410 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3f2806ad00 0x7f3f2806d1c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c034000 tx=0x7f3f3c02e000 comp rx=0 tx=0).stop 2024-09-13T16:44:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.945+0000 7f3f4e800640 1 -- 172.21.15.28:0/875945410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f400500c0 msgr2=0x7f3f4016f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.945+0000 7f400d200640 1 -- 172.21.15.28:0/4041695902 shutdown_connections 2024-09-13T16:44:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.945+0000 7f400d200640 1 --2- 172.21.15.28:0/4041695902 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3fe806ad00 0x7f3fe806d1c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.945+0000 7f400d200640 1 --2- 172.21.15.28:0/4041695902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40000d8970 0x7f40000db7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.945+0000 7f400d200640 1 --2- 172.21.15.28:0/4041695902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40000503e0 0x7f40000dcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.945+0000 7f400d200640 1 --2- 172.21.15.28:0/4041695902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40000072f0 0x7f400016f7f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.945+0000 7f400d200640 1 -- 172.21.15.28:0/4041695902 >> 172.21.15.28:0/4041695902 conn(0x7f4000054040 msgr2=0x7f4000050a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.945+0000 7f3f4e800640 1 --2- 172.21.15.28:0/875945410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f400500c0 0x7f3f4016f810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3f4802c9a0 tx=0x7f3f48067310 comp rx=0 tx=0).stop 2024-09-13T16:44:03.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.945+0000 7f400d200640 1 -- 172.21.15.28:0/4041695902 shutdown_connections 2024-09-13T16:44:03.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.945+0000 7f400d200640 1 -- 172.21.15.28:0/4041695902 wait complete. 2024-09-13T16:44:03.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.946+0000 7f3f4e800640 1 -- 172.21.15.28:0/875945410 shutdown_connections 2024-09-13T16:44:03.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.946+0000 7f3f4e800640 1 --2- 172.21.15.28:0/875945410 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3f2806ad00 0x7f3f2806d1c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.946+0000 7f3f4e800640 1 --2- 172.21.15.28:0/875945410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f400dccb0 0x7f3f4004dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.946+0000 7f3f4e800640 1 --2- 172.21.15.28:0/875945410 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f40050600 0x7f3f400d8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.946+0000 7f3f4e800640 1 --2- 172.21.15.28:0/875945410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f400500c0 0x7f3f4016f810 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.946+0000 7f3f4e800640 1 -- 172.21.15.28:0/875945410 >> 172.21.15.28:0/875945410 conn(0x7f3f40054050 msgr2=0x7f3f40048f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.946+0000 7f3f4e800640 1 -- 172.21.15.28:0/875945410 shutdown_connections 2024-09-13T16:44:03.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.946+0000 7f3f4e800640 1 -- 172.21.15.28:0/875945410 wait complete. 2024-09-13T16:44:03.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.946+0000 7f4bcfe00640 1 -- 172.21.15.28:0/3659288185 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4bc804e460 con 0x7f4bc80502a0 2024-09-13T16:44:03.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.947+0000 7f2cde600640 1 -- 172.21.15.28:0/1356275622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2cd00462f0 con 0x7f2cd00059d0 2024-09-13T16:44:03.950 INFO:teuthology.orchestra.run.smithi028.stdout:55834574851 2024-09-13T16:44:03.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.947+0000 7f4bc6a00640 1 -- 172.21.15.28:0/3659288185 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4bbc0a9070 con 0x7f4bc80502a0 2024-09-13T16:44:03.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.948+0000 7f2cd4c00640 1 -- 172.21.15.28:0/1356275622 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2cd80bf020 con 0x7f2cd00059d0 2024-09-13T16:44:03.951 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-13T16:44:03.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.950+0000 7f4bcfe00640 1 -- 172.21.15.28:0/3659288185 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4bb006acd0 msgr2=0x7f4bb006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.951+0000 7f4bcfe00640 1 --2- 172.21.15.28:0/3659288185 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4bb006acd0 0x7f4bb006d190 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009c00 tx=0x7f4bc0009190 comp rx=0 tx=0).stop 2024-09-13T16:44:03.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.951+0000 7f4bcfe00640 1 -- 172.21.15.28:0/3659288185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bc80502a0 msgr2=0x7f4bc81741e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.951+0000 7f4bcfe00640 1 --2- 172.21.15.28:0/3659288185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bc80502a0 0x7f4bc81741e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc031000 tx=0x7f4bbc0046c0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.951+0000 7f4bcfe00640 1 -- 172.21.15.28:0/3659288185 shutdown_connections 2024-09-13T16:44:03.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.951+0000 7f4bcfe00640 1 --2- 172.21.15.28:0/3659288185 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4bb006acd0 0x7f4bb006d190 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.951+0000 7f4bcfe00640 1 --2- 172.21.15.28:0/3659288185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bc80d8840 0x7f4bc8174720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.951+0000 7f4bcfe00640 1 --2- 172.21.15.28:0/3659288185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bc80502a0 0x7f4bc81741e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.951+0000 7f4bcfe00640 1 --2- 172.21.15.28:0/3659288185 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc8049190 0x7f4bc80d0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.951+0000 7f4bcfe00640 1 -- 172.21.15.28:0/3659288185 >> 172.21.15.28:0/3659288185 conn(0x7f4bc8054420 msgr2=0x7f4bc8006840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.952+0000 7f4bcfe00640 1 -- 172.21.15.28:0/3659288185 shutdown_connections 2024-09-13T16:44:03.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.952+0000 7f4bcfe00640 1 -- 172.21.15.28:0/3659288185 wait complete. 2024-09-13T16:44:03.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.952+0000 7f2cde600640 1 -- 172.21.15.28:0/1356275622 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f2cb806ac60 msgr2=0x7f2cb806d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.952+0000 7f2cde600640 1 --2- 172.21.15.28:0/1356275622 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f2cb806ac60 0x7f2cb806d120 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0045c30 tx=0x7f2cc4008040 comp rx=0 tx=0).stop 2024-09-13T16:44:03.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.953+0000 7f2cde600640 1 -- 172.21.15.28:0/1356275622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd00059d0 msgr2=0x7f2cd00465d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.953+0000 7f2cde600640 1 --2- 172.21.15.28:0/1356275622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd00059d0 0x7f2cd00465d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8074ff0 tx=0x7f2cd8075230 comp rx=0 tx=0).stop 2024-09-13T16:44:03.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.954+0000 7f2cde600640 1 -- 172.21.15.28:0/1356275622 shutdown_connections 2024-09-13T16:44:03.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.954+0000 7f2cde600640 1 --2- 172.21.15.28:0/1356275622 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f2cb806ac60 0x7f2cb806d120 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.954+0000 7f2cde600640 1 --2- 172.21.15.28:0/1356275622 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cd0050740 0x7f2cd00bf770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.954+0000 7f2cde600640 1 --2- 172.21.15.28:0/1356275622 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cd0034220 0x7f2cd0046f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.954+0000 7f2cde600640 1 --2- 172.21.15.28:0/1356275622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd00059d0 0x7f2cd00465d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.954+0000 7f2cde600640 1 -- 172.21.15.28:0/1356275622 >> 172.21.15.28:0/1356275622 conn(0x7f2cd004fc20 msgr2=0x7f2cd00d3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.954+0000 7f2cde600640 1 -- 172.21.15.28:0/1356275622 shutdown_connections 2024-09-13T16:44:03.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.954+0000 7f2cde600640 1 -- 172.21.15.28:0/1356275622 wait complete. 2024-09-13T16:44:03.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.964+0000 7eff81400640 1 -- 172.21.15.28:0/914183604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7eff7c0c9280 con 0x7eff7c156b80 2024-09-13T16:44:03.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.965+0000 7eff73e00640 1 -- 172.21.15.28:0/914183604 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7eff74057b50 con 0x7eff7c156b80 2024-09-13T16:44:03.967 INFO:teuthology.orchestra.run.smithi028.stdout:55834574851 2024-09-13T16:44:03.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.970+0000 7eff81400640 1 -- 172.21.15.28:0/914183604 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7eff5c06ad00 msgr2=0x7eff5c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.970+0000 7eff81400640 1 --2- 172.21.15.28:0/914183604 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7eff5c06ad00 0x7eff5c06d1c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7eff6c000f60 tx=0x7eff6c009040 comp rx=0 tx=0).stop 2024-09-13T16:44:03.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.970+0000 7eff81400640 1 -- 172.21.15.28:0/914183604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff7c156b80 msgr2=0x7eff7c0c2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.970+0000 7eff81400640 1 --2- 172.21.15.28:0/914183604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff7c156b80 0x7eff7c0c2140 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7eff7400c6d0 tx=0x7eff7400cb90 comp rx=0 tx=0).stop 2024-09-13T16:44:03.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.970+0000 7f54c1600640 1 -- 172.21.15.28:0/301792768 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f54940050c0 con 0x7f54bc0cb560 2024-09-13T16:44:03.973 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.2 2024-09-13T16:44:03.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.971+0000 7eff81400640 1 -- 172.21.15.28:0/914183604 shutdown_connections 2024-09-13T16:44:03.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.971+0000 7eff81400640 1 --2- 172.21.15.28:0/914183604 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7eff5c06ad00 0x7eff5c06d1c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.971+0000 7eff81400640 1 --2- 172.21.15.28:0/914183604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff7c156b80 0x7eff7c0c2140 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.971+0000 7eff81400640 1 --2- 172.21.15.28:0/914183604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7eff7c06bfc0 0x7eff7c0c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.971+0000 7eff81400640 1 --2- 172.21.15.28:0/914183604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7eff7c06bbe0 0x7eff7c0c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.971+0000 7eff81400640 1 -- 172.21.15.28:0/914183604 >> 172.21.15.28:0/914183604 conn(0x7eff7c0ca2f0 msgr2=0x7eff7c151460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.971+0000 7f54b2a00640 1 -- 172.21.15.28:0/301792768 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f54a8057ee0 con 0x7f54bc0cb560 2024-09-13T16:44:03.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.971+0000 7eff81400640 1 -- 172.21.15.28:0/914183604 shutdown_connections 2024-09-13T16:44:03.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.971+0000 7eff81400640 1 -- 172.21.15.28:0/914183604 wait complete. 2024-09-13T16:44:03.974 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-13T16:44:03.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.973+0000 7f1e66000640 1 -- 172.21.15.28:0/3247433571 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f1e38005500 con 0x7f1e600c3660 2024-09-13T16:44:03.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.974+0000 7f1e57400640 1 -- 172.21.15.28:0/3247433571 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1e50057bd0 con 0x7f1e600c3660 2024-09-13T16:44:03.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.974+0000 7f54c1600640 1 -- 172.21.15.28:0/301792768 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f549c06ac50 msgr2=0x7f549c06d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.974+0000 7f54c1600640 1 --2- 172.21.15.28:0/301792768 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f549c06ac50 0x7f549c06d110 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f54ac02a240 tx=0x7f54ac02e000 comp rx=0 tx=0).stop 2024-09-13T16:44:03.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.974+0000 7f54c1600640 1 -- 172.21.15.28:0/301792768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54bc0cb560 msgr2=0x7f54bc0b3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.974+0000 7f54c1600640 1 --2- 172.21.15.28:0/301792768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54bc0cb560 0x7f54bc0b3ef0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f54a800d5f0 tx=0x7f54a800dab0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.976 INFO:teuthology.orchestra.run.smithi028.stdout:55834574851 2024-09-13T16:44:03.977 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.6 2024-09-13T16:44:03.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.975+0000 7f54c1600640 1 -- 172.21.15.28:0/301792768 shutdown_connections 2024-09-13T16:44:03.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.975+0000 7f54c1600640 1 --2- 172.21.15.28:0/301792768 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f549c06ac50 0x7f549c06d110 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.975+0000 7f54c1600640 1 --2- 172.21.15.28:0/301792768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc152f20 0x7f54bc1fc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.975+0000 7f54c1600640 1 --2- 172.21.15.28:0/301792768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54bc0cb560 0x7f54bc0b3ef0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.975+0000 7f54c1600640 1 --2- 172.21.15.28:0/301792768 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54bc0ca5d0 0x7f54bc14ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.975+0000 7f54c1600640 1 -- 172.21.15.28:0/301792768 >> 172.21.15.28:0/301792768 conn(0x7f54bc065c50 msgr2=0x7f54bc151a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.975+0000 7f54c1600640 1 -- 172.21.15.28:0/301792768 shutdown_connections 2024-09-13T16:44:03.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.975+0000 7f54c1600640 1 -- 172.21.15.28:0/301792768 wait complete. 2024-09-13T16:44:03.980 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.0 2024-09-13T16:44:03.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.978+0000 7f1e66000640 1 -- 172.21.15.28:0/3247433571 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1e4006ac00 msgr2=0x7f1e4006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.978+0000 7f1e66000640 1 --2- 172.21.15.28:0/3247433571 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1e4006ac00 0x7f1e4006d0c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1e6014db70 tx=0x7f1e4c02e000 comp rx=0 tx=0).stop 2024-09-13T16:44:03.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.978+0000 7f1e66000640 1 -- 172.21.15.28:0/3247433571 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e600c3660 msgr2=0x7f1e600cca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.978+0000 7f1e66000640 1 --2- 172.21.15.28:0/3247433571 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e600c3660 0x7f1e600cca50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1e5000c5e0 tx=0x7f1e5000caa0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.979+0000 7f1e66000640 1 -- 172.21.15.28:0/3247433571 shutdown_connections 2024-09-13T16:44:03.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.979+0000 7f1e66000640 1 --2- 172.21.15.28:0/3247433571 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1e4006ac00 0x7f1e4006d0c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.979+0000 7f1e66000640 1 --2- 172.21.15.28:0/3247433571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e60152f50 0x7f1e6015b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.979+0000 7f1e66000640 1 --2- 172.21.15.28:0/3247433571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e600c9b00 0x7f1e60148b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.979+0000 7f1e66000640 1 --2- 172.21.15.28:0/3247433571 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e600c3660 0x7f1e600cca50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.979+0000 7f1e66000640 1 -- 172.21.15.28:0/3247433571 >> 172.21.15.28:0/3247433571 conn(0x7f1e60071790 msgr2=0x7f1e601e2a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.979+0000 7f1e66000640 1 -- 172.21.15.28:0/3247433571 shutdown_connections 2024-09-13T16:44:03.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.980+0000 7f1e66000640 1 -- 172.21.15.28:0/3247433571 wait complete. 2024-09-13T16:44:03.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.981+0000 7f7d53800640 1 -- 172.21.15.28:0/2314503442 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f7d24005500 con 0x7f7d4c157110 2024-09-13T16:44:03.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.982+0000 7f7d49600640 1 -- 172.21.15.28:0/2314503442 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7d34040090 con 0x7f7d4c157110 2024-09-13T16:44:03.983 INFO:teuthology.orchestra.run.smithi028.stdout:55834574851 2024-09-13T16:44:03.985 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.1 2024-09-13T16:44:03.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.984+0000 7f7d53800640 1 -- 172.21.15.28:0/2314503442 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7d2806ac30 msgr2=0x7f7d2806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.984+0000 7f7d53800640 1 --2- 172.21.15.28:0/2314503442 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7d2806ac30 0x7f7d2806d0f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c006f40 tx=0x7f7d3c006e90 comp rx=0 tx=0).stop 2024-09-13T16:44:03.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.984+0000 7f7d53800640 1 -- 172.21.15.28:0/2314503442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d4c157110 msgr2=0x7f7d4c1595a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:03.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.984+0000 7f7d53800640 1 --2- 172.21.15.28:0/2314503442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d4c157110 0x7f7d4c1595a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d34031000 tx=0x7f7d34004a40 comp rx=0 tx=0).stop 2024-09-13T16:44:03.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.984+0000 7f7d53800640 1 -- 172.21.15.28:0/2314503442 shutdown_connections 2024-09-13T16:44:03.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.984+0000 7f7d53800640 1 --2- 172.21.15.28:0/2314503442 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7d2806ac30 0x7f7d2806d0f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.984+0000 7f7d53800640 1 --2- 172.21.15.28:0/2314503442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7d4c1574f0 0x7f7d4c1f27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.984+0000 7f7d53800640 1 --2- 172.21.15.28:0/2314503442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7d4c157110 0x7f7d4c1595a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.984+0000 7f7d53800640 1 --2- 172.21.15.28:0/2314503442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d4c06d880 0x7f7d4c158530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:03.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.984+0000 7f7d53800640 1 -- 172.21.15.28:0/2314503442 >> 172.21.15.28:0/2314503442 conn(0x7f7d4c0ca6b0 msgr2=0x7f7d4c0c3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:03.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.984+0000 7f7d53800640 1 -- 172.21.15.28:0/2314503442 shutdown_connections 2024-09-13T16:44:03.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:03.985+0000 7f7d53800640 1 -- 172.21.15.28:0/2314503442 wait complete. 2024-09-13T16:44:03.995 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.3 2024-09-13T16:44:04.003 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.4 2024-09-13T16:44:04.005 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.5 2024-09-13T16:44:04.011 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.7 2024-09-13T16:44:04.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T16:44:04.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T16:44:04.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T16:44:04.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T16:44:04.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T16:44:05.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T16:44:05.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T16:44:05.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T16:44:05.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.225+0000 7f670f200640 1 Processor -- start 2024-09-13T16:44:05.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.225+0000 7f670f200640 1 -- start start 2024-09-13T16:44:05.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.226+0000 7f98a7200640 1 Processor -- start 2024-09-13T16:44:05.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.226+0000 7f98a7200640 1 -- start start 2024-09-13T16:44:05.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.228+0000 7f98a7200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f98a8152e50 0x7f98a80caec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.228+0000 7f98a7200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98a80ca420 0x7f98a80cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.228+0000 7f98a7200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98a80c9230 0x7f98a80c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.228+0000 7f98a7200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98a806aa90 con 0x7f98a80ca420 2024-09-13T16:44:05.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.228+0000 7f98a7200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98a8158a80 con 0x7f98a8152e50 2024-09-13T16:44:05.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.228+0000 7f98a7200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a80c9c00 con 0x7f98a80c9230 2024-09-13T16:44:05.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.228+0000 7f98a5e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f98a8152e50 0x7f98a80caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.229+0000 7f98a5e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f98a8152e50 0x7f98a80caec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60828/0 (socket says 172.21.15.28:60828) 2024-09-13T16:44:05.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.229+0000 7f98a5e00640 1 -- 172.21.15.28:0/2986649901 learned_addr learned my addr 172.21.15.28:0/2986649901 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:05.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.229+0000 7f670f200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f671006bbe0 0x7f67100c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.229+0000 7f98a6800640 1 --2- 172.21.15.28:0/2986649901 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98a80c9230 0x7f98a80c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.230+0000 7f670f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6710070370 con 0x7f671006bfc0 2024-09-13T16:44:05.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.230+0000 7f670f200640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67101609e0 con 0x7f6710156b80 2024-09-13T16:44:05.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.230+0000 7f670f200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6710160cc0 con 0x7f671006bbe0 2024-09-13T16:44:05.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.229+0000 7f98a5e00640 1 -- 172.21.15.28:0/2986649901 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98a80c9230 msgr2=0x7f98a80c96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:05.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.229+0000 7f98a5e00640 1 --2- 172.21.15.28:0/2986649901 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98a80c9230 0x7f98a80c96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.229+0000 7f98a5e00640 1 -- 172.21.15.28:0/2986649901 >> v2:172.21.15.28:3300/0 conn(0x7f98a80ca420 msgr2=0x7f98a80cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:05.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.229+0000 7f98a5e00640 1 --2- 172.21.15.28:0/2986649901 >> v2:172.21.15.28:3300/0 conn(0x7f98a80ca420 0x7f98a80cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.229+0000 7f98a5e00640 1 -- 172.21.15.28:0/2986649901 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a8153230 con 0x7f98a8152e50 2024-09-13T16:44:05.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.230+0000 7f670e800640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f6710156b80 0x7f6710070dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:51536/0 (socket says 172.21.15.28:51536) 2024-09-13T16:44:05.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.230+0000 7f670e800640 1 -- 172.21.15.28:0/1549243927 learned_addr learned my addr 172.21.15.28:0/1549243927 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:05.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.229+0000 7f670de00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f671006bbe0 0x7f67100c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.231+0000 7f670de00640 1 -- 172.21.15.28:0/1549243927 >> v1:172.21.15.191:6789/0 conn(0x7f6710156b80 legacy=0x7f6710070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:05.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.231+0000 7f670de00640 1 -- 172.21.15.28:0/1549243927 >> v1:172.21.15.28:6789/0 conn(0x7f671006bfc0 legacy=0x7f6710160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:05.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.231+0000 7f670de00640 1 -- 172.21.15.28:0/1549243927 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6710160fa0 con 0x7f671006bbe0 2024-09-13T16:44:05.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.231+0000 7f98a5e00640 1 --2- 172.21.15.28:0/2986649901 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98a8152e50 0x7f98a80caec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f98a80c91c0 tx=0x7f9894032ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b240d320b2d266a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.232+0000 7f98a4a00640 1 -- 172.21.15.28:0/2986649901 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9894036870 con 0x7f98a8152e50 2024-09-13T16:44:05.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.232+0000 7f98a4a00640 1 -- 172.21.15.28:0/2986649901 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98940369d0 con 0x7f98a8152e50 2024-09-13T16:44:05.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.233+0000 7f670de00640 1 --2- 172.21.15.28:0/1549243927 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f671006bbe0 0x7f67100c67e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6708009dc0 tx=0x7f670802a630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ab687a6602f2fbf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.232+0000 7f98a4a00640 1 -- 172.21.15.28:0/2986649901 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9894036da0 con 0x7f98a8152e50 2024-09-13T16:44:05.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.233+0000 7f98a7200640 1 -- 172.21.15.28:0/2986649901 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98a8152e50 msgr2=0x7f98a80caec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.233+0000 7f98a7200640 1 --2- 172.21.15.28:0/2986649901 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98a8152e50 0x7f98a80caec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f98a80c91c0 tx=0x7f9894032ab0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.234+0000 7f98a7200640 1 -- 172.21.15.28:0/2986649901 shutdown_connections 2024-09-13T16:44:05.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.234+0000 7f98a7200640 1 --2- 172.21.15.28:0/2986649901 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98a80c9230 0x7f98a80c96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.234+0000 7f98a7200640 1 --2- 172.21.15.28:0/2986649901 >> v2:172.21.15.28:3300/0 conn(0x7f98a80ca420 0x7f98a80cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.234+0000 7f98a7200640 1 --2- 172.21.15.28:0/2986649901 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98a8152e50 0x7f98a80caec0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.234+0000 7f98a7200640 1 -- 172.21.15.28:0/2986649901 >> 172.21.15.28:0/2986649901 conn(0x7f98a8065c50 msgr2=0x7f98a8066060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:05.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.234+0000 7f98a7200640 1 -- 172.21.15.28:0/2986649901 shutdown_connections 2024-09-13T16:44:05.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.234+0000 7f670ca00640 1 -- 172.21.15.28:0/1549243927 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6708007740 con 0x7f671006bbe0 2024-09-13T16:44:05.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.234+0000 7f670ca00640 1 -- 172.21.15.28:0/1549243927 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670802d050 con 0x7f671006bbe0 2024-09-13T16:44:05.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.235+0000 7f670ca00640 1 -- 172.21.15.28:0/1549243927 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6708007ac0 con 0x7f671006bbe0 2024-09-13T16:44:05.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.235+0000 7f670f200640 1 -- 172.21.15.28:0/1549243927 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f671006bbe0 msgr2=0x7f67100c67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.234+0000 7f98a7200640 1 -- 172.21.15.28:0/2986649901 wait complete. 2024-09-13T16:44:05.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.235+0000 7f670f200640 1 --2- 172.21.15.28:0/1549243927 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f671006bbe0 0x7f67100c67e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6708009dc0 tx=0x7f670802a630 comp rx=0 tx=0).stop 2024-09-13T16:44:05.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.236+0000 7f670f200640 1 -- 172.21.15.28:0/1549243927 shutdown_connections 2024-09-13T16:44:05.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.236+0000 7f670f200640 1 --2- 172.21.15.28:0/1549243927 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f671006bbe0 0x7f67100c67e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.236+0000 7f670f200640 1 -- 172.21.15.28:0/1549243927 >> 172.21.15.28:0/1549243927 conn(0x7f67100ca2f0 msgr2=0x7f67100ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:05.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.237+0000 7f670f200640 1 -- 172.21.15.28:0/1549243927 shutdown_connections 2024-09-13T16:44:05.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.237+0000 7f98a7200640 1 Processor -- start 2024-09-13T16:44:05.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.237+0000 7f670f200640 1 -- 172.21.15.28:0/1549243927 wait complete. 2024-09-13T16:44:05.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.237+0000 7f98a7200640 1 -- start start 2024-09-13T16:44:05.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.239+0000 7f98a7200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98a80c9230 0x7f98a81edda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.239+0000 7f98a7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a80ca420 0x7f98a80b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.239+0000 7f98a7200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98a8152e50 0x7f98a81553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.238+0000 7f3852600640 1 Processor -- start 2024-09-13T16:44:05.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.239+0000 7f3852600640 1 -- start start 2024-09-13T16:44:05.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.240+0000 7f670f200640 1 Processor -- start 2024-09-13T16:44:05.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.240+0000 7f98a7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a8154330 con 0x7f98a80ca420 2024-09-13T16:44:05.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.241+0000 7f670f200640 1 -- start start 2024-09-13T16:44:05.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.240+0000 7f98a7200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a8153e00 con 0x7f98a8152e50 2024-09-13T16:44:05.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.240+0000 7f98a7200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a81ed910 con 0x7f98a80c9230 2024-09-13T16:44:05.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.240+0000 7f98a5e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98a80c9230 0x7f98a81edda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.240+0000 7f98a6800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98a8152e50 0x7f98a81553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.240+0000 7f98a6800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98a8152e50 0x7f98a81553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60842/0 (socket says 172.21.15.28:60842) 2024-09-13T16:44:05.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.240+0000 7f98a6800640 1 -- 172.21.15.28:0/944102191 learned_addr learned my addr 172.21.15.28:0/944102191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:05.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.241+0000 7f98a6800640 1 -- 172.21.15.28:0/944102191 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98a80c9230 msgr2=0x7f98a81edda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.241+0000 7f98a6800640 1 --2- 172.21.15.28:0/944102191 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98a80c9230 0x7f98a81edda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.241+0000 7f98a6800640 1 -- 172.21.15.28:0/944102191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a80ca420 msgr2=0x7f98a80b38b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:44:05.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.241+0000 7f3852600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f384404aa10 0x7f38440059d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.241+0000 7f98a5400640 1 --2- 172.21.15.28:0/944102191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a80ca420 0x7f98a80b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.242+0000 7f98a6800640 1 --2- 172.21.15.28:0/944102191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a80ca420 0x7f98a80b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.242+0000 7f3851200640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f3844044b00 0x7f3844046360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:51546/0 (socket says 172.21.15.28:51546) 2024-09-13T16:44:05.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.242+0000 7f3851c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3844044ee0 0x7f38440ff800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59574/0 (socket says 172.21.15.28:59574) 2024-09-13T16:44:05.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.242+0000 7f98a6800640 1 -- 172.21.15.28:0/944102191 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98940094d0 con 0x7f98a8152e50 2024-09-13T16:44:05.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.242+0000 7f98a6800640 1 --2- 172.21.15.28:0/944102191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98a8152e50 0x7f98a81553d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f98a000efc0 tx=0x7f98a000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.242+0000 7f989e000640 1 -- 172.21.15.28:0/944102191 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a0014480 con 0x7f98a8152e50 2024-09-13T16:44:05.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.242+0000 7f989e000640 1 -- 172.21.15.28:0/944102191 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a000f040 con 0x7f98a8152e50 2024-09-13T16:44:05.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.242+0000 7f3852600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3844045630 con 0x7f3844044ee0 2024-09-13T16:44:05.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.242+0000 7f3852600640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3844045ed0 con 0x7f3844044b00 2024-09-13T16:44:05.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.242+0000 7f3852600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f384404b3d0 con 0x7f384404aa10 2024-09-13T16:44:05.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.242+0000 7f384be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f384404aa10 0x7f38440059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.242+0000 7f384be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f384404aa10 0x7f38440059d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41682/0 (socket says 172.21.15.28:41682) 2024-09-13T16:44:05.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.243+0000 7f384be00640 1 -- 172.21.15.28:0/2654265897 learned_addr learned my addr 172.21.15.28:0/2654265897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:05.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.243+0000 7f98a7200640 1 -- 172.21.15.28:0/944102191 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a8154050 con 0x7f98a8152e50 2024-09-13T16:44:05.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.243+0000 7f989e000640 1 -- 172.21.15.28:0/944102191 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a0014820 con 0x7f98a8152e50 2024-09-13T16:44:05.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.243+0000 7f670f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f671006bbe0 0x7f67101f28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.243+0000 7f98a7200640 1 -- 172.21.15.28:0/944102191 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a81ed5a0 con 0x7f98a8152e50 2024-09-13T16:44:05.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.243+0000 7f670f200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f671006bfc0 0x7f67100c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.243+0000 7f670f200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6710156b80 0x7f6710158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.244+0000 7f670d400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f671006bfc0 0x7f67100c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.244+0000 7f670d400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f671006bfc0 0x7f67100c3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60844/0 (socket says 172.21.15.28:60844) 2024-09-13T16:44:05.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.243+0000 7f384b400640 1 -- 172.21.15.28:0/2654265897 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112043378 0 0) 0x7f3844045ed0 con 0x7f3844044b00 2024-09-13T16:44:05.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.244+0000 7f384b400640 1 -- 172.21.15.28:0/2654265897 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3834005730 con 0x7f3844044b00 2024-09-13T16:44:05.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.244+0000 7f670f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67101f5ed0 con 0x7f671006bbe0 2024-09-13T16:44:05.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.244+0000 7f670f200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67101f61b0 con 0x7f671006bfc0 2024-09-13T16:44:05.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.244+0000 7f670f200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67101f6490 con 0x7f6710156b80 2024-09-13T16:44:05.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.244+0000 7f384b400640 1 -- 172.21.15.28:0/2654265897 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790148057 0 0) 0x7f3844045630 con 0x7f3844044ee0 2024-09-13T16:44:05.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.244+0000 7f384b400640 1 -- 172.21.15.28:0/2654265897 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3844045ed0 con 0x7f3844044ee0 2024-09-13T16:44:05.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.245+0000 7f384be00640 1 -- 172.21.15.28:0/2654265897 >> v1:172.21.15.191:6789/0 conn(0x7f3844044b00 legacy=0x7f3844046360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.245+0000 7f384be00640 1 -- 172.21.15.28:0/2654265897 >> v1:172.21.15.28:6789/0 conn(0x7f3844044ee0 legacy=0x7f38440ff800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.245+0000 7f384be00640 1 -- 172.21.15.28:0/2654265897 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38440f6a40 con 0x7f384404aa10 2024-09-13T16:44:05.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.245+0000 7f670e800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6710156b80 0x7f6710158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.245+0000 7f670d400640 1 -- 172.21.15.28:0/2235657249 learned_addr learned my addr 172.21.15.28:0/2235657249 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:05.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.245+0000 7f670de00640 1 --2- 172.21.15.28:0/2235657249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f671006bbe0 0x7f67101f28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.245+0000 7f670d400640 1 -- 172.21.15.28:0/2235657249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6710156b80 msgr2=0x7f6710158130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.245+0000 7f670d400640 1 --2- 172.21.15.28:0/2235657249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6710156b80 0x7f6710158130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.245+0000 7f670d400640 1 -- 172.21.15.28:0/2235657249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f671006bbe0 msgr2=0x7f67101f28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.245+0000 7f670d400640 1 --2- 172.21.15.28:0/2235657249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f671006bbe0 0x7f67101f28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.245+0000 7f670d400640 1 -- 172.21.15.28:0/2235657249 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6708009a70 con 0x7f671006bfc0 2024-09-13T16:44:05.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.246+0000 7f670d400640 1 --2- 172.21.15.28:0/2235657249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f671006bfc0 0x7f67100c3bd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f670000c410 tx=0x7f670000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.246+0000 7f989e000640 1 -- 172.21.15.28:0/944102191 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f98a0014980 con 0x7f98a8152e50 2024-09-13T16:44:05.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.247+0000 7f384be00640 1 --2- 172.21.15.28:0/2654265897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f384404aa10 0x7f38440059d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3838009870 tx=0x7f383802a3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad82e2bc21de7d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.247+0000 7f6706000640 1 -- 172.21.15.28:0/2235657249 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67000035f0 con 0x7f671006bfc0 2024-09-13T16:44:05.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.247+0000 7f98a7200640 1 -- 172.21.15.28:0/944102191 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a8159810 con 0x7f98a8152e50 2024-09-13T16:44:05.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.247+0000 7f6706000640 1 -- 172.21.15.28:0/2235657249 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6700003750 con 0x7f671006bfc0 2024-09-13T16:44:05.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.247+0000 7f6706000640 1 -- 172.21.15.28:0/2235657249 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6700003b40 con 0x7f671006bfc0 2024-09-13T16:44:05.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.248+0000 7f670f200640 1 -- 172.21.15.28:0/2235657249 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67101f6770 con 0x7f671006bfc0 2024-09-13T16:44:05.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.248+0000 7f670f200640 1 -- 172.21.15.28:0/2235657249 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671006e310 con 0x7f671006bfc0 2024-09-13T16:44:05.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.248+0000 7f989e000640 1 --2- 172.21.15.28:0/944102191 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f988806ac30 0x7f988806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.248+0000 7f98a5e00640 1 --2- 172.21.15.28:0/944102191 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f988806ac30 0x7f988806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.249+0000 7f384b400640 1 -- 172.21.15.28:0/2654265897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838032640 con 0x7f384404aa10 2024-09-13T16:44:05.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.249+0000 7f384b400640 1 -- 172.21.15.28:0/2654265897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38380327a0 con 0x7f384404aa10 2024-09-13T16:44:05.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.249+0000 7f384b400640 1 -- 172.21.15.28:0/2654265897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838032b70 con 0x7f384404aa10 2024-09-13T16:44:05.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.249+0000 7f989e000640 1 -- 172.21.15.28:0/944102191 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98a0089f90 con 0x7f98a8152e50 2024-09-13T16:44:05.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.250+0000 7f670f200640 1 -- 172.21.15.28:0/2235657249 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67100c67e0 con 0x7f671006bfc0 2024-09-13T16:44:05.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.250+0000 7f3852600640 1 -- 172.21.15.28:0/2654265897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f384404aa10 msgr2=0x7f38440059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.250+0000 7f3852600640 1 --2- 172.21.15.28:0/2654265897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f384404aa10 0x7f38440059d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3838009870 tx=0x7f383802a3e0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.250+0000 7f3852600640 1 -- 172.21.15.28:0/2654265897 shutdown_connections 2024-09-13T16:44:05.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.250+0000 7f3852600640 1 --2- 172.21.15.28:0/2654265897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f384404aa10 0x7f38440059d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.250+0000 7f3852600640 1 -- 172.21.15.28:0/2654265897 >> 172.21.15.28:0/2654265897 conn(0x7f384404fc20 msgr2=0x7f3844050030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:05.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.250+0000 7f3852600640 1 -- 172.21.15.28:0/2654265897 shutdown_connections 2024-09-13T16:44:05.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.250+0000 7f3852600640 1 -- 172.21.15.28:0/2654265897 wait complete. 2024-09-13T16:44:05.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.250+0000 7f98a5e00640 1 --2- 172.21.15.28:0/944102191 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f988806ac30 0x7f988806d0f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f989403d000 tx=0x7f9894008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.252+0000 7f3852600640 1 Processor -- start 2024-09-13T16:44:05.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.252+0000 7f3852600640 1 -- start start 2024-09-13T16:44:05.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.253+0000 7f989e000640 1 -- 172.21.15.28:0/944102191 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f98a0055040 con 0x7f98a8152e50 2024-09-13T16:44:05.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.254+0000 7f3852600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3844044b00 0x7f384419e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.254+0000 7f3852600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3844044ee0 0x7f38440f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.254+0000 7f3852600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f384404aa10 0x7f38440f3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.254+0000 7f3852600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38440fd360 con 0x7f384404aa10 2024-09-13T16:44:05.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.254+0000 7f3852600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38440f9060 con 0x7f3844044b00 2024-09-13T16:44:05.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.254+0000 7f3852600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384419dba0 con 0x7f3844044ee0 2024-09-13T16:44:05.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.254+0000 7f3851200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3844044b00 0x7f384419e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.254+0000 7f3851200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3844044b00 0x7f384419e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60850/0 (socket says 172.21.15.28:60850) 2024-09-13T16:44:05.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.254+0000 7f3851200640 1 -- 172.21.15.28:0/1680971245 learned_addr learned my addr 172.21.15.28:0/1680971245 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:05.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.254+0000 7f384be00640 1 --2- 172.21.15.28:0/1680971245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3844044ee0 0x7f38440f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.255+0000 7f3851200640 1 -- 172.21.15.28:0/1680971245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3844044ee0 msgr2=0x7f38440f2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.255+0000 7f3851200640 1 --2- 172.21.15.28:0/1680971245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3844044ee0 0x7f38440f2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.255+0000 7f3851200640 1 -- 172.21.15.28:0/1680971245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f384404aa10 msgr2=0x7f38440f3280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:44:05.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.255+0000 7f3851c00640 1 --2- 172.21.15.28:0/1680971245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f384404aa10 0x7f38440f3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.256+0000 7f6706000640 1 -- 172.21.15.28:0/2235657249 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6700003ca0 con 0x7f671006bfc0 2024-09-13T16:44:05.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.256+0000 7f3851200640 1 --2- 172.21.15.28:0/1680971245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f384404aa10 0x7f38440f3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.256+0000 7f3851200640 1 -- 172.21.15.28:0/1680971245 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38380094d0 con 0x7f3844044b00 2024-09-13T16:44:05.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.257+0000 7f3851c00640 1 --2- 172.21.15.28:0/1680971245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f384404aa10 0x7f38440f3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:44:05.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.257+0000 7f3851200640 1 --2- 172.21.15.28:0/1680971245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3844044b00 0x7f384419e380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f384c074920 tx=0x7f384c067a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.257+0000 7f3848c00640 1 -- 172.21.15.28:0/1680971245 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c06f830 con 0x7f3844044b00 2024-09-13T16:44:05.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.258+0000 7f3848c00640 1 -- 172.21.15.28:0/1680971245 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384c06ae90 con 0x7f3844044b00 2024-09-13T16:44:05.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.258+0000 7f3848c00640 1 -- 172.21.15.28:0/1680971245 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c06fc70 con 0x7f3844044b00 2024-09-13T16:44:05.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.258+0000 7f6706000640 1 --2- 172.21.15.28:0/2235657249 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f66f006ac30 0x7f66f006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.258+0000 7f6706000640 1 -- 172.21.15.28:0/2235657249 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6700088590 con 0x7f671006bfc0 2024-09-13T16:44:05.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.258+0000 7f3852600640 1 -- 172.21.15.28:0/1680971245 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38440fa780 con 0x7f3844044b00 2024-09-13T16:44:05.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.258+0000 7f3852600640 1 -- 172.21.15.28:0/1680971245 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38440f9590 con 0x7f3844044b00 2024-09-13T16:44:05.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.260+0000 7f670de00640 1 --2- 172.21.15.28:0/2235657249 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f66f006ac30 0x7f66f006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.260+0000 7f6706000640 1 -- 172.21.15.28:0/2235657249 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6700053900 con 0x7f671006bfc0 2024-09-13T16:44:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.261+0000 7f3852600640 1 -- 172.21.15.28:0/1680971245 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3828005250 con 0x7f3844044b00 2024-09-13T16:44:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.261+0000 7f670de00640 1 --2- 172.21.15.28:0/2235657249 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f66f006ac30 0x7f66f006d0f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6708034000 tx=0x7f670802f000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.262+0000 7f3848c00640 1 -- 172.21.15.28:0/1680971245 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f384c082400 con 0x7f3844044b00 2024-09-13T16:44:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.264+0000 7f3848c00640 1 --2- 172.21.15.28:0/1680971245 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f382c06ad00 0x7f382c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.264+0000 7f3848c00640 1 -- 172.21.15.28:0/1680971245 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f384c0f1fd0 con 0x7f3844044b00 2024-09-13T16:44:05.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.265+0000 7f384be00640 1 --2- 172.21.15.28:0/1680971245 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f382c06ad00 0x7f382c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.266+0000 7f384be00640 1 --2- 172.21.15.28:0/1680971245 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f382c06ad00 0x7f382c06d1c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3838034000 tx=0x7f383802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.267+0000 7f3848c00640 1 -- 172.21.15.28:0/1680971245 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f384c0bd2d0 con 0x7f3844044b00 2024-09-13T16:44:05.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.309+0000 7f4746200640 1 Processor -- start 2024-09-13T16:44:05.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.310+0000 7f4746200640 1 -- start start 2024-09-13T16:44:05.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.311+0000 7f4746200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4740157110 0x7f47400c5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.312+0000 7f4746200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f474006df90 0x7f47400c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.312+0000 7f4746200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474015dee0 con 0x7f47401574f0 2024-09-13T16:44:05.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.312+0000 7f4746200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4740071c60 con 0x7f474006df90 2024-09-13T16:44:05.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.312+0000 7f4746200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47400c1ff0 con 0x7f4740157110 2024-09-13T16:44:05.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.313+0000 7f473ea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f47401574f0 0x7f474006d880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59582/0 (socket says 172.21.15.28:59582) 2024-09-13T16:44:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.313+0000 7f473ea00640 1 -- 172.21.15.28:0/3717830282 learned_addr learned my addr 172.21.15.28:0/3717830282 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.313+0000 7f473f400640 1 --2- 172.21.15.28:0/3717830282 >> v2:172.21.15.191:3300/0 conn(0x7f4740157110 0x7f47400c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.313+0000 7f473fe00640 1 --2- 172.21.15.28:0/3717830282 >> v2:172.21.15.28:3300/0 conn(0x7f474006df90 0x7f47400c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.314+0000 7f473f400640 1 -- 172.21.15.28:0/3717830282 >> v2:172.21.15.28:3300/0 conn(0x7f474006df90 msgr2=0x7f47400c5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.314+0000 7f473e000640 1 -- 172.21.15.28:0/3717830282 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392229598 0 0) 0x7f474015dee0 con 0x7f47401574f0 2024-09-13T16:44:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.314+0000 7f473f400640 1 --2- 172.21.15.28:0/3717830282 >> v2:172.21.15.28:3300/0 conn(0x7f474006df90 0x7f47400c5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.314+0000 7f473f400640 1 -- 172.21.15.28:0/3717830282 >> v1:172.21.15.28:6789/0 conn(0x7f47401574f0 legacy=0x7f474006d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.314+0000 7f473f400640 1 -- 172.21.15.28:0/3717830282 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474006d3f0 con 0x7f4740157110 2024-09-13T16:44:05.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.315+0000 7f473fe00640 1 --2- 172.21.15.28:0/3717830282 >> v2:172.21.15.28:3300/0 conn(0x7f474006df90 0x7f47400c5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:05.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.316+0000 7f744e200640 1 Processor -- start 2024-09-13T16:44:05.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.316+0000 7f744e200640 1 -- start start 2024-09-13T16:44:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.316+0000 7f473f400640 1 --2- 172.21.15.28:0/3717830282 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4740157110 0x7f47400c5580 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f474006ca00 tx=0x7f473002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bed6cb0f140271ed server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.317+0000 7f473e000640 1 -- 172.21.15.28:0/3717830282 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4730032440 con 0x7f4740157110 2024-09-13T16:44:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.317+0000 7f473e000640 1 -- 172.21.15.28:0/3717830282 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47300325a0 con 0x7f4740157110 2024-09-13T16:44:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.317+0000 7f473e000640 1 -- 172.21.15.28:0/3717830282 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4730032930 con 0x7f4740157110 2024-09-13T16:44:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.318+0000 7f4746200640 1 -- 172.21.15.28:0/3717830282 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4740157110 msgr2=0x7f47400c5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.318+0000 7f4746200640 1 --2- 172.21.15.28:0/3717830282 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4740157110 0x7f47400c5580 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f474006ca00 tx=0x7f473002a1a0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.318+0000 7f4746200640 1 -- 172.21.15.28:0/3717830282 shutdown_connections 2024-09-13T16:44:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.318+0000 7f4746200640 1 --2- 172.21.15.28:0/3717830282 >> v2:172.21.15.28:3300/0 conn(0x7f474006df90 0x7f47400c5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.318+0000 7f4746200640 1 --2- 172.21.15.28:0/3717830282 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4740157110 0x7f47400c5580 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.318+0000 7f744e200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74480b3020 0x7f74480c4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.318+0000 7f744e200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f74480ca260 0x7f74480b1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.318+0000 7f744e200640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74480ca7a0 con 0x7f74480b3400 2024-09-13T16:44:05.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.318+0000 7f744e200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f744815de50 con 0x7f74480ca260 2024-09-13T16:44:05.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.318+0000 7f744e200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74480cadd0 con 0x7f74480b3020 2024-09-13T16:44:05.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.317+0000 7f132de00640 1 Processor -- start 2024-09-13T16:44:05.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.317+0000 7f132de00640 1 -- start start 2024-09-13T16:44:05.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.319+0000 7f7446a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f74480ca260 0x7f74480b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.318+0000 7f4746200640 1 -- 172.21.15.28:0/3717830282 >> 172.21.15.28:0/3717830282 conn(0x7f47400ca6b0 msgr2=0x7f47400caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:05.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.319+0000 7f7446a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f74480ca260 0x7f74480b1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60872/0 (socket says 172.21.15.28:60872) 2024-09-13T16:44:05.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.319+0000 7f7446a00640 1 -- 172.21.15.28:0/987079561 learned_addr learned my addr 172.21.15.28:0/987079561 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:05.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.319+0000 7f7447400640 1 --2- 172.21.15.28:0/987079561 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74480b3020 0x7f74480c4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.319+0000 7f4746200640 1 -- 172.21.15.28:0/3717830282 shutdown_connections 2024-09-13T16:44:05.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.320+0000 7f132de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f132815dd00 0x7f13280c9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.320+0000 7f7446a00640 1 -- 172.21.15.28:0/987079561 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74480b3020 msgr2=0x7f74480c4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:05.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.320+0000 7f7446a00640 1 --2- 172.21.15.28:0/987079561 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74480b3020 0x7f74480c4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.320+0000 7f7446a00640 1 -- 172.21.15.28:0/987079561 >> v1:172.21.15.191:6789/0 conn(0x7f74480b3400 legacy=0x7f74480c4de0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-13T16:44:05.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.320+0000 7f7446a00640 1 -- 172.21.15.28:0/987079561 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f744815ce30 con 0x7f74480ca260 2024-09-13T16:44:05.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.321+0000 7f132de00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13280c9b70 0x7f13280caae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.321+0000 7f132de00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13281606a0 con 0x7f1328157310 2024-09-13T16:44:05.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.321+0000 7f132de00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13280ca690 con 0x7f132815dd00 2024-09-13T16:44:05.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.321+0000 7f132de00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f132815e100 con 0x7f13280c9b70 2024-09-13T16:44:05.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.321+0000 7f7446a00640 1 --2- 172.21.15.28:0/987079561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74480ca260 0x7f74480b1430 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7440009ea0 tx=0x7f744002a810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f6da0522ebcfe6d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.322+0000 7f1326000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f132815dd00 0x7f13280c9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.322+0000 7f1326000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f132815dd00 0x7f13280c9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41726/0 (socket says 172.21.15.28:41726) 2024-09-13T16:44:05.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.322+0000 7f1326000640 1 -- 172.21.15.28:0/4276007141 learned_addr learned my addr 172.21.15.28:0/4276007141 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:05.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.322+0000 7f7446000640 1 -- 172.21.15.28:0/987079561 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7440033610 con 0x7f74480ca260 2024-09-13T16:44:05.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.323+0000 7f7446000640 1 -- 172.21.15.28:0/987079561 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7440033770 con 0x7f74480ca260 2024-09-13T16:44:05.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.323+0000 7f7446000640 1 -- 172.21.15.28:0/987079561 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7440033b00 con 0x7f74480ca260 2024-09-13T16:44:05.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.323+0000 7f1326000640 1 -- 172.21.15.28:0/4276007141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13280c9b70 msgr2=0x7f13280caae0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-13T16:44:05.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.323+0000 7f391b200640 1 Processor -- start 2024-09-13T16:44:05.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.323+0000 7f1326000640 1 --2- 172.21.15.28:0/4276007141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13280c9b70 0x7f13280caae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.323+0000 7f1326000640 1 -- 172.21.15.28:0/4276007141 >> v1:172.21.15.191:6789/0 conn(0x7f1328157310 legacy=0x7f13280c35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:05.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.320+0000 7f4746200640 1 -- 172.21.15.28:0/3717830282 wait complete. 2024-09-13T16:44:05.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.323+0000 7f4746200640 1 Processor -- start 2024-09-13T16:44:05.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.323+0000 7f7447400640 1 --2- 172.21.15.28:0/987079561 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74480b3020 0x7f74480c4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:05.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.323+0000 7f1326000640 1 -- 172.21.15.28:0/4276007141 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13280c3ce0 con 0x7f132815dd00 2024-09-13T16:44:05.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.323+0000 7f391b200640 1 -- start start 2024-09-13T16:44:05.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.324+0000 7f744e200640 1 -- 172.21.15.28:0/987079561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74480ca260 msgr2=0x7f74480b1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.324+0000 7f744e200640 1 --2- 172.21.15.28:0/987079561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74480ca260 0x7f74480b1430 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7440009ea0 tx=0x7f744002a810 comp rx=0 tx=0).stop 2024-09-13T16:44:05.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f391b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3914152f50 0x7f3914159690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.323+0000 7f4746200640 1 -- start start 2024-09-13T16:44:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f4746200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f474006df90 0x7f474015e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f4746200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4740157110 0x7f474015e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f4746200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47401574f0 0x7f47401fabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f4746200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474015ab60 con 0x7f47401574f0 2024-09-13T16:44:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f4746200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4740153040 con 0x7f474006df90 2024-09-13T16:44:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f4746200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474015b450 con 0x7f4740157110 2024-09-13T16:44:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f473ea00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4740157110 0x7f474015e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f391b200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39140cae30 0x7f39140cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f391b200640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3914159bd0 con 0x7f39140c3660 2024-09-13T16:44:05.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f391b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39140c3f60 con 0x7f3914152f50 2024-09-13T16:44:05.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f391b200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f391415d0d0 con 0x7f39140cae30 2024-09-13T16:44:05.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f744e200640 1 -- 172.21.15.28:0/987079561 shutdown_connections 2024-09-13T16:44:05.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f1326000640 1 --2- 172.21.15.28:0/4276007141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132815dd00 0x7f13280c9630 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f132815e240 tx=0x7f131802a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a70b980b382a4cb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f744e200640 1 --2- 172.21.15.28:0/987079561 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74480ca260 0x7f74480b1430 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f3913400640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f39140c3660 0x7f39140c95c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:51568/0 (socket says 172.21.15.28:51568) 2024-09-13T16:44:05.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f3913400640 1 -- 172.21.15.28:0/894586514 learned_addr learned my addr 172.21.15.28:0/894586514 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:05.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.325+0000 7f473ea00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4740157110 0x7f474015e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37672/0 (socket says 172.21.15.28:37672) 2024-09-13T16:44:05.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.326+0000 7f473ea00640 1 -- 172.21.15.28:0/3261140187 learned_addr learned my addr 172.21.15.28:0/3261140187 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:05.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.326+0000 7f3913e00640 1 --2- 172.21.15.28:0/894586514 >> v2:172.21.15.28:3300/0 conn(0x7f3914152f50 0x7f3914159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.326+0000 7f744e200640 1 --2- 172.21.15.28:0/987079561 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74480b3020 0x7f74480c4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.326+0000 7f744e200640 1 -- 172.21.15.28:0/987079561 >> 172.21.15.28:0/987079561 conn(0x7f74480666a0 msgr2=0x7f74480ce5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:05.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.326+0000 7f1325600640 1 -- 172.21.15.28:0/4276007141 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1318031600 con 0x7f132815dd00 2024-09-13T16:44:05.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.326+0000 7f3912a00640 1 -- 172.21.15.28:0/894586514 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722967947 0 0) 0x7f3900001110 con 0x7f39140c3660 2024-09-13T16:44:05.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.326+0000 7f3912a00640 1 -- 172.21.15.28:0/894586514 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f8003520 con 0x7f39140c3660 2024-09-13T16:44:05.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.326+0000 7f1325600640 1 -- 172.21.15.28:0/4276007141 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131802d050 con 0x7f132815dd00 2024-09-13T16:44:05.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.326+0000 7fb207e00640 1 Processor -- start 2024-09-13T16:44:05.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.326+0000 7f3919200640 1 --2- 172.21.15.28:0/894586514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39140cae30 0x7f39140cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.326+0000 7f1325600640 1 -- 172.21.15.28:0/4276007141 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13180319c0 con 0x7f132815dd00 2024-09-13T16:44:05.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f3912a00640 1 -- 172.21.15.28:0/894586514 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1888394689 0 0) 0x7f38f8003520 con 0x7f39140c3660 2024-09-13T16:44:05.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f3912a00640 1 -- 172.21.15.28:0/894586514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39140cae30 msgr2=0x7f39140cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:05.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f3912a00640 1 --2- 172.21.15.28:0/894586514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39140cae30 0x7f39140cb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f3912a00640 1 -- 172.21.15.28:0/894586514 >> v2:172.21.15.28:3300/0 conn(0x7f3914152f50 msgr2=0x7f3914159690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:05.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f3912a00640 1 --2- 172.21.15.28:0/894586514 >> v2:172.21.15.28:3300/0 conn(0x7f3914152f50 0x7f3914159690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f3912a00640 1 -- 172.21.15.28:0/894586514 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39140cb800 con 0x7f39140c3660 2024-09-13T16:44:05.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f744e200640 1 -- 172.21.15.28:0/987079561 shutdown_connections 2024-09-13T16:44:05.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f473f400640 1 --2- 172.21.15.28:0/3261140187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f474006df90 0x7f474015e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f744e200640 1 -- 172.21.15.28:0/987079561 wait complete. 2024-09-13T16:44:05.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f473fe00640 1 --2- 172.21.15.28:0/3261140187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47401574f0 0x7f47401fabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f473ea00640 1 -- 172.21.15.28:0/3261140187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f474006df90 msgr2=0x7f474015e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f473ea00640 1 --2- 172.21.15.28:0/3261140187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f474006df90 0x7f474015e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f473ea00640 1 -- 172.21.15.28:0/3261140187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47401574f0 msgr2=0x7f47401fabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f473ea00640 1 --2- 172.21.15.28:0/3261140187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47401574f0 0x7f47401fabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f3912a00640 1 -- 172.21.15.28:0/894586514 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1654382301 0 0) 0x7f3900001800 con 0x7f39140c3660 2024-09-13T16:44:05.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f3912a00640 1 -- 172.21.15.28:0/894586514 >> v1:172.21.15.191:6789/0 conn(0x7f39140c3660 legacy=0x7f39140c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f3912a00640 1 --2- 172.21.15.28:0/894586514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38f8003970 0x7f38f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f132de00640 1 -- 172.21.15.28:0/4276007141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132815dd00 msgr2=0x7f13280c9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f132de00640 1 --2- 172.21.15.28:0/4276007141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132815dd00 0x7f13280c9630 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f132815e240 tx=0x7f131802a570 comp rx=0 tx=0).stop 2024-09-13T16:44:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7f473ea00640 1 -- 172.21.15.28:0/3261140187 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c009400 con 0x7f4740157110 2024-09-13T16:44:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.328+0000 7f473fe00640 1 --2- 172.21.15.28:0/3261140187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47401574f0 0x7f47401fabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:44:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.328+0000 7f473f400640 1 --2- 172.21.15.28:0/3261140187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f474006df90 0x7f474015e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:44:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.328+0000 7f3912a00640 1 --2- 172.21.15.28:0/894586514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39140cae30 0x7f38f80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.328+0000 7f3912a00640 1 --2- 172.21.15.28:0/894586514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38f8004d00 0x7f38f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.328+0000 7f3912a00640 1 -- 172.21.15.28:0/894586514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39140c3f60 con 0x7f38f8004d00 2024-09-13T16:44:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.328+0000 7f3912a00640 1 -- 172.21.15.28:0/894586514 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391415d0d0 con 0x7f38f8003970 2024-09-13T16:44:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.328+0000 7f3912a00640 1 -- 172.21.15.28:0/894586514 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f800bab0 con 0x7f39140cae30 2024-09-13T16:44:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.327+0000 7fb207e00640 1 -- start start 2024-09-13T16:44:05.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.328+0000 7f3913400640 1 --2- 172.21.15.28:0/894586514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39140cae30 0x7f38f80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.329+0000 7f132de00640 1 -- 172.21.15.28:0/4276007141 shutdown_connections 2024-09-13T16:44:05.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.329+0000 7f132de00640 1 --2- 172.21.15.28:0/4276007141 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13280c9b70 0x7f13280caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.329+0000 7f132de00640 1 --2- 172.21.15.28:0/4276007141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132815dd00 0x7f13280c9630 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.329+0000 7f132de00640 1 -- 172.21.15.28:0/4276007141 >> 172.21.15.28:0/4276007141 conn(0x7f1328071790 msgr2=0x7f132806b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:05.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.328+0000 7f473ea00640 1 --2- 172.21.15.28:0/3261140187 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4740157110 0x7f474015e870 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f473800c890 tx=0x7f473800cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.329+0000 7f3913400640 1 -- 172.21.15.28:0/894586514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38f8003970 msgr2=0x7f38f8003d90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-13T16:44:05.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.329+0000 7f3913400640 1 --2- 172.21.15.28:0/894586514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38f8003970 0x7f38f8003d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.329+0000 7f3913400640 1 -- 172.21.15.28:0/894586514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38f8004d00 msgr2=0x7f38f8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:05.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.329+0000 7f3913400640 1 --2- 172.21.15.28:0/894586514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38f8004d00 0x7f38f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.329+0000 7f3913400640 1 -- 172.21.15.28:0/894586514 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f8003520 con 0x7f39140cae30 2024-09-13T16:44:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.329+0000 7f4737400640 1 -- 172.21.15.28:0/3261140187 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473800a640 con 0x7f4740157110 2024-09-13T16:44:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.329+0000 7f4737400640 1 -- 172.21.15.28:0/3261140187 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473800a7c0 con 0x7f4740157110 2024-09-13T16:44:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.329+0000 7f4737400640 1 -- 172.21.15.28:0/3261140187 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473800ab70 con 0x7f4740157110 2024-09-13T16:44:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.329+0000 7f4746200640 1 -- 172.21.15.28:0/3261140187 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4730009400 con 0x7f4740157110 2024-09-13T16:44:05.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.329+0000 7f4746200640 1 -- 172.21.15.28:0/3261140187 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4740153290 con 0x7f4740157110 2024-09-13T16:44:05.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.330+0000 7f744e200640 1 Processor -- start 2024-09-13T16:44:05.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.330+0000 7f132de00640 1 -- 172.21.15.28:0/4276007141 shutdown_connections 2024-09-13T16:44:05.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.330+0000 7fb207e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb2000d8840 0x7fb200049920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.330+0000 7fb207e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2000dee10 0x7fb200049e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.330+0000 7fb207e00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb200050820 con 0x7fb2000502a0 2024-09-13T16:44:05.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.330+0000 7fb207e00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2000e28b0 con 0x7fb2000d8840 2024-09-13T16:44:05.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.330+0000 7fb207e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2000e1c30 con 0x7fb2000dee10 2024-09-13T16:44:05.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.330+0000 7fb206a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb2000d8840 0x7fb200049920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.330+0000 7fb206000640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fb2000502a0 0x7fb2000de700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:51584/0 (socket says 172.21.15.28:51584) 2024-09-13T16:44:05.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.330+0000 7fb206a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb2000d8840 0x7fb200049920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60902/0 (socket says 172.21.15.28:60902) 2024-09-13T16:44:05.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.330+0000 7fb206000640 1 -- 172.21.15.28:0/950021376 learned_addr learned my addr 172.21.15.28:0/950021376 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:05.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.330+0000 7fb207400640 1 --2- 172.21.15.28:0/950021376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2000dee10 0x7fb200049e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.331+0000 7fb206a00640 1 -- 172.21.15.28:0/950021376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2000dee10 msgr2=0x7fb200049e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:05.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.331+0000 7fb206a00640 1 --2- 172.21.15.28:0/950021376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2000dee10 0x7fb200049e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.331+0000 7fb205600640 1 -- 172.21.15.28:0/950021376 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265740789 0 0) 0x7fb200050820 con 0x7fb2000502a0 2024-09-13T16:44:05.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.331+0000 7fb206a00640 1 -- 172.21.15.28:0/950021376 >> v1:172.21.15.191:6789/0 conn(0x7fb2000502a0 legacy=0x7fb2000de700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.331+0000 7fb206a00640 1 -- 172.21.15.28:0/950021376 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2000e1f10 con 0x7fb2000d8840 2024-09-13T16:44:05.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.331+0000 7fb207400640 1 --2- 172.21.15.28:0/950021376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2000dee10 0x7fb200049e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:05.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.331+0000 7f744e200640 1 -- start start 2024-09-13T16:44:05.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.332+0000 7f4746200640 1 -- 172.21.15.28:0/3261140187 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4740153480 con 0x7f4740157110 2024-09-13T16:44:05.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.332+0000 7f4737400640 1 -- 172.21.15.28:0/3261140187 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f473800acd0 con 0x7f4740157110 2024-09-13T16:44:05.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.332+0000 7f132de00640 1 -- 172.21.15.28:0/4276007141 wait complete. 2024-09-13T16:44:05.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.332+0000 7f3913400640 1 --2- 172.21.15.28:0/894586514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39140cae30 0x7f38f80047c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f390000c110 tx=0x7f38f80057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30a474aa373d6c55 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.332+0000 7fb206a00640 1 --2- 172.21.15.28:0/950021376 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2000d8840 0x7fb200049920 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb1f80097a0 tx=0x7fb1f802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18525625a9e2b396 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.333+0000 7f744e200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74480b3020 0x7f74481e4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.333+0000 7f744e200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74480b3400 0x7f74481520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.333+0000 7fb205600640 1 -- 172.21.15.28:0/950021376 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8032440 con 0x7fb2000d8840 2024-09-13T16:44:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.333+0000 7fb205600640 1 -- 172.21.15.28:0/950021376 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f80325a0 con 0x7fb2000d8840 2024-09-13T16:44:05.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.333+0000 7fb205600640 1 -- 172.21.15.28:0/950021376 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8032930 con 0x7fb2000d8840 2024-09-13T16:44:05.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.333+0000 7f744e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74480ca260 0x7f74481e5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.333+0000 7f7446a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74480b3400 0x7f74481520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.333+0000 7f4737400640 1 --2- 172.21.15.28:0/3261140187 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f472006ad00 0x7f472006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.333+0000 7f4737400640 1 -- 172.21.15.28:0/3261140187 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4738088d40 con 0x7f4740157110 2024-09-13T16:44:05.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.333+0000 7f7446a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74480b3400 0x7f74481520d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60916/0 (socket says 172.21.15.28:60916) 2024-09-13T16:44:05.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.334+0000 7f7447400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74480b3020 0x7f74481e4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.334+0000 7f7447400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74480b3020 0x7f74481e4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37702/0 (socket says 172.21.15.28:37702) 2024-09-13T16:44:05.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.334+0000 7f7447e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74480ca260 0x7f74481e5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.334+0000 7f473f400640 1 --2- 172.21.15.28:0/3261140187 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f472006ad00 0x7f472006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.334+0000 7f7447e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74480ca260 0x7f74481e5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41758/0 (socket says 172.21.15.28:41758) 2024-09-13T16:44:05.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.334+0000 7fb207e00640 1 -- 172.21.15.28:0/950021376 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2000d8840 msgr2=0x7fb200049920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.334+0000 7fb207e00640 1 --2- 172.21.15.28:0/950021376 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2000d8840 0x7fb200049920 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb1f80097a0 tx=0x7fb1f802a180 comp rx=0 tx=0).stop 2024-09-13T16:44:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.334+0000 7f7446a00640 1 -- 172.21.15.28:0/403522969 learned_addr learned my addr 172.21.15.28:0/403522969 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.334+0000 7f744e200640 1 -- 172.21.15.28:0/403522969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74481e4210 con 0x7f74480ca260 2024-09-13T16:44:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.334+0000 7f744e200640 1 -- 172.21.15.28:0/403522969 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74480c7fc0 con 0x7f74480b3400 2024-09-13T16:44:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.334+0000 7f744e200640 1 -- 172.21.15.28:0/403522969 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74480c86f0 con 0x7f74480b3020 2024-09-13T16:44:05.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.334+0000 7f132de00640 1 Processor -- start 2024-09-13T16:44:05.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.334+0000 7f132de00640 1 -- start start 2024-09-13T16:44:05.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.334+0000 7fb207e00640 1 -- 172.21.15.28:0/950021376 shutdown_connections 2024-09-13T16:44:05.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.334+0000 7fb207e00640 1 --2- 172.21.15.28:0/950021376 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2000dee10 0x7fb200049e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.334+0000 7fb207e00640 1 --2- 172.21.15.28:0/950021376 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2000d8840 0x7fb200049920 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.334+0000 7fb207e00640 1 -- 172.21.15.28:0/950021376 >> 172.21.15.28:0/950021376 conn(0x7fb200054420 msgr2=0x7fb200054830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:05.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.335+0000 7fb207e00640 1 -- 172.21.15.28:0/950021376 shutdown_connections 2024-09-13T16:44:05.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.335+0000 7f7447400640 1 -- 172.21.15.28:0/403522969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74480b3400 msgr2=0x7f74481520d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.335+0000 7f7447400640 1 --2- 172.21.15.28:0/403522969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74480b3400 0x7f74481520d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.335+0000 7f7447400640 1 -- 172.21.15.28:0/403522969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74480ca260 msgr2=0x7f74481e5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.335+0000 7f7447400640 1 --2- 172.21.15.28:0/403522969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74480ca260 0x7f74481e5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.335+0000 7f7447400640 1 -- 172.21.15.28:0/403522969 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7438009400 con 0x7f74480b3020 2024-09-13T16:44:05.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.335+0000 7f7446a00640 1 --2- 172.21.15.28:0/403522969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74480b3400 0x7f74481520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:05.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.335+0000 7f7447400640 1 --2- 172.21.15.28:0/403522969 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74480b3020 0x7f74481e4ee0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f743800e900 tx=0x7f743800edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.335+0000 7f473f400640 1 --2- 172.21.15.28:0/3261140187 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f472006ad00 0x7f472006d1c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4730034000 tx=0x7f473002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.335+0000 7fb207e00640 1 -- 172.21.15.28:0/950021376 wait complete. 2024-09-13T16:44:05.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.336+0000 7f132de00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13280c9b70 0x7f13280c78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.336+0000 7f132de00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1328157310 0x7f13281511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.336+0000 7f132de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132815dd00 0x7f13280ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.336+0000 7f1326a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13280c9b70 0x7f13280c78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.336+0000 7f1326a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13280c9b70 0x7f13280c78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37718/0 (socket says 172.21.15.28:37718) 2024-09-13T16:44:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.337+0000 7f1327400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132815dd00 0x7f13280ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.337+0000 7f1327400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132815dd00 0x7f13280ce8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41772/0 (socket says 172.21.15.28:41772) 2024-09-13T16:44:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.337+0000 7f132de00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1328151df0 con 0x7f132815dd00 2024-09-13T16:44:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.337+0000 7f132de00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13280c7e00 con 0x7f1328157310 2024-09-13T16:44:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.337+0000 7f743f400640 1 -- 172.21.15.28:0/403522969 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743800ca00 con 0x7f74480b3020 2024-09-13T16:44:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.337+0000 7f743f400640 1 -- 172.21.15.28:0/403522969 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743800cb60 con 0x7f74480b3020 2024-09-13T16:44:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.337+0000 7f743f400640 1 -- 172.21.15.28:0/403522969 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7438013470 con 0x7f74480b3020 2024-09-13T16:44:05.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.337+0000 7f132de00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13280cb320 con 0x7f13280c9b70 2024-09-13T16:44:05.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.337+0000 7f1326a00640 1 -- 172.21.15.28:0/990304764 learned_addr learned my addr 172.21.15.28:0/990304764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:05.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.337+0000 7f1326a00640 1 -- 172.21.15.28:0/990304764 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1328157310 msgr2=0x7f13281511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.337+0000 7f1326a00640 1 --2- 172.21.15.28:0/990304764 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1328157310 0x7f13281511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.337+0000 7f1326a00640 1 -- 172.21.15.28:0/990304764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132815dd00 msgr2=0x7f13280ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.337+0000 7f1326a00640 1 --2- 172.21.15.28:0/990304764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132815dd00 0x7f13280ce8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.337+0000 7f1326a00640 1 -- 172.21.15.28:0/990304764 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13180094d0 con 0x7f13280c9b70 2024-09-13T16:44:05.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.338+0000 7f1326a00640 1 --2- 172.21.15.28:0/990304764 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13280c9b70 0x7f13280c78c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f131400c480 tx=0x7f131400c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.338+0000 7fb207e00640 1 Processor -- start 2024-09-13T16:44:05.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.338+0000 7f3912a00640 1 -- 172.21.15.28:0/894586514 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3900005ad0 con 0x7f39140cae30 2024-09-13T16:44:05.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.338+0000 7f3912a00640 1 -- 172.21.15.28:0/894586514 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390002d070 con 0x7f39140cae30 2024-09-13T16:44:05.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.338+0000 7f3912a00640 1 -- 172.21.15.28:0/894586514 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3900007070 con 0x7f39140cae30 2024-09-13T16:44:05.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.338+0000 7f4737400640 1 -- 172.21.15.28:0/3261140187 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4740153480 con 0x7f4740157110 2024-09-13T16:44:05.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.337+0000 7f744e200640 1 -- 172.21.15.28:0/403522969 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7440009b30 con 0x7f74480b3020 2024-09-13T16:44:05.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.338+0000 7f744e200640 1 -- 172.21.15.28:0/403522969 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74481e71b0 con 0x7f74480b3020 2024-09-13T16:44:05.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.339+0000 7f131f400640 1 -- 172.21.15.28:0/990304764 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13140052d0 con 0x7f13280c9b70 2024-09-13T16:44:05.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.339+0000 7f131f400640 1 -- 172.21.15.28:0/990304764 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1314005430 con 0x7f13280c9b70 2024-09-13T16:44:05.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.339+0000 7f131f400640 1 -- 172.21.15.28:0/990304764 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1314012870 con 0x7f13280c9b70 2024-09-13T16:44:05.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.339+0000 7fb207e00640 1 -- start start 2024-09-13T16:44:05.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.340+0000 7f391b200640 1 -- 172.21.15.28:0/894586514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39140cae30 msgr2=0x7f38f80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.340+0000 7f744e200640 1 -- 172.21.15.28:0/403522969 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7420005250 con 0x7f74480b3020 2024-09-13T16:44:05.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.340+0000 7f391b200640 1 --2- 172.21.15.28:0/894586514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39140cae30 0x7f38f80047c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f390000c110 tx=0x7f38f80057f0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.340+0000 7f3919200640 1 -- 172.21.15.28:0/894586514 reap_dead start 2024-09-13T16:44:05.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.340+0000 7fb207e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2000502a0 0x7fb2000db570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.341+0000 7fb207e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2000d8840 0x7fb2000ce420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.341+0000 7fb207e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2000dee10 0x7fb200052560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.341+0000 7fb207e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2001746b0 con 0x7fb2000d8840 2024-09-13T16:44:05.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.341+0000 7fb207e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb200174990 con 0x7fb2000dee10 2024-09-13T16:44:05.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.341+0000 7fb207e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb200174c70 con 0x7fb2000502a0 2024-09-13T16:44:05.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.341+0000 7fb207400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2000dee10 0x7fb200052560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.342+0000 7fb207400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2000dee10 0x7fb200052560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60942/0 (socket says 172.21.15.28:60942) 2024-09-13T16:44:05.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.342+0000 7fb207400640 1 -- 172.21.15.28:0/1273436901 learned_addr learned my addr 172.21.15.28:0/1273436901 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:05.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.342+0000 7fb206a00640 1 --2- 172.21.15.28:0/1273436901 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2000502a0 0x7fb2000db570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.342+0000 7fb206000640 1 --2- 172.21.15.28:0/1273436901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2000d8840 0x7fb2000ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.341+0000 7f743f400640 1 -- 172.21.15.28:0/403522969 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f74380135d0 con 0x7f74480b3020 2024-09-13T16:44:05.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.343+0000 7f743f400640 1 --2- 172.21.15.28:0/403522969 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f742806ac80 0x7f742806d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.343+0000 7f743f400640 1 -- 172.21.15.28:0/403522969 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7438089450 con 0x7f74480b3020 2024-09-13T16:44:05.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.344+0000 7f7446a00640 1 --2- 172.21.15.28:0/403522969 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f742806ac80 0x7f742806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.340+0000 7f132de00640 1 -- 172.21.15.28:0/990304764 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1328151b10 con 0x7f13280c9b70 2024-09-13T16:44:05.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.340+0000 7f132de00640 1 -- 172.21.15.28:0/990304764 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13281509e0 con 0x7f13280c9b70 2024-09-13T16:44:05.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.342+0000 7f132de00640 1 -- 172.21.15.28:0/990304764 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1328067460 con 0x7f13280c9b70 2024-09-13T16:44:05.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.343+0000 7fb207400640 1 -- 172.21.15.28:0/1273436901 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2000502a0 msgr2=0x7fb2000db570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.343+0000 7fb207400640 1 --2- 172.21.15.28:0/1273436901 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2000502a0 0x7fb2000db570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.343+0000 7fb207400640 1 -- 172.21.15.28:0/1273436901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2000d8840 msgr2=0x7fb2000ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.343+0000 7fb207400640 1 --2- 172.21.15.28:0/1273436901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2000d8840 0x7fb2000ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.343+0000 7fb207400640 1 -- 172.21.15.28:0/1273436901 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f8009400 con 0x7fb2000dee10 2024-09-13T16:44:05.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.343+0000 7fb206000640 1 --2- 172.21.15.28:0/1273436901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2000d8840 0x7fb2000ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:05.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.344+0000 7fb207400640 1 --2- 172.21.15.28:0/1273436901 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2000dee10 0x7fb200052560 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb20804e4e0 tx=0x7fb208067310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.344+0000 7fb206a00640 1 --2- 172.21.15.28:0/1273436901 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2000502a0 0x7fb2000db570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:05.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.344+0000 7fb1fea00640 1 -- 172.21.15.28:0/1273436901 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208071760 con 0x7fb2000dee10 2024-09-13T16:44:05.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.344+0000 7fb1fea00640 1 -- 172.21.15.28:0/1273436901 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20806ae90 con 0x7fb2000dee10 2024-09-13T16:44:05.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.344+0000 7fb1fea00640 1 -- 172.21.15.28:0/1273436901 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208071b20 con 0x7fb2000dee10 2024-09-13T16:44:05.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.344+0000 7fb207e00640 1 -- 172.21.15.28:0/1273436901 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb200174f50 con 0x7fb2000dee10 2024-09-13T16:44:05.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.344+0000 7fb207e00640 1 -- 172.21.15.28:0/1273436901 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2000d95f0 con 0x7fb2000dee10 2024-09-13T16:44:05.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.345+0000 7f7446a00640 1 --2- 172.21.15.28:0/403522969 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f742806ac80 0x7f742806d140 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7440000f80 tx=0x7f7440009000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.343+0000 7f131f400640 1 -- 172.21.15.28:0/990304764 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f13140129d0 con 0x7f13280c9b70 2024-09-13T16:44:05.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.344+0000 7f131f400640 1 --2- 172.21.15.28:0/990304764 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f130806ad00 0x7f130806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.345+0000 7f131f400640 1 -- 172.21.15.28:0/990304764 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1314088de0 con 0x7f13280c9b70 2024-09-13T16:44:05.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.345+0000 7f1326000640 1 --2- 172.21.15.28:0/990304764 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f130806ad00 0x7f130806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.346+0000 7f1326000640 1 --2- 172.21.15.28:0/990304764 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f130806ad00 0x7f130806d1c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1318033000 tx=0x7f131802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.345+0000 7f391b200640 1 -- 172.21.15.28:0/894586514 shutdown_connections 2024-09-13T16:44:05.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.345+0000 7f391b200640 1 -- 172.21.15.28:0/894586514 >> 172.21.15.28:0/894586514 conn(0x7f3914071790 msgr2=0x7f391406b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:05.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.346+0000 7fb1fea00640 1 -- 172.21.15.28:0/1273436901 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb208090070 con 0x7fb2000dee10 2024-09-13T16:44:05.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.346+0000 7fb207e00640 1 -- 172.21.15.28:0/1273436901 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2000d97e0 con 0x7fb2000dee10 2024-09-13T16:44:05.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.347+0000 7f391b200640 1 -- 172.21.15.28:0/894586514 shutdown_connections 2024-09-13T16:44:05.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.347+0000 7f743f400640 1 -- 172.21.15.28:0/403522969 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7438054750 con 0x7f74480b3020 2024-09-13T16:44:05.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.347+0000 7f391b200640 1 -- 172.21.15.28:0/894586514 wait complete. 2024-09-13T16:44:05.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.348+0000 7fb1fea00640 1 --2- 172.21.15.28:0/1273436901 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb1e806ac00 0x7fb1e806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.348+0000 7fb1fea00640 1 -- 172.21.15.28:0/1273436901 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2081034a0 con 0x7fb2000dee10 2024-09-13T16:44:05.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.348+0000 7fb206a00640 1 --2- 172.21.15.28:0/1273436901 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb1e806ac00 0x7fb1e806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.349+0000 7fb206a00640 1 --2- 172.21.15.28:0/1273436901 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb1e806ac00 0x7fb1e806d0c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8034000 tx=0x7fb1f802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.350+0000 7f391b200640 1 Processor -- start 2024-09-13T16:44:05.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.350+0000 7f391b200640 1 -- start start 2024-09-13T16:44:05.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.352+0000 7f131f400640 1 -- 172.21.15.28:0/990304764 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1314054030 con 0x7f13280c9b70 2024-09-13T16:44:05.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.352+0000 7fb1fea00640 1 -- 172.21.15.28:0/1273436901 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb2080d21b0 con 0x7fb2000dee10 2024-09-13T16:44:05.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.352+0000 7f391b200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39140cae30 0x7f39140b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.353+0000 7f3913e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39140cae30 0x7f39140b2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.353+0000 7f3913e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39140cae30 0x7f39140b2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37728/0 (socket says 172.21.15.28:37728) 2024-09-13T16:44:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.352+0000 7f391b200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39140ce2b0 0x7f39140b3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.353+0000 7f391b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39141f6190 0x7f39141f6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.353+0000 7f391b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39140b28a0 con 0x7f39141f6190 2024-09-13T16:44:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.353+0000 7f391b200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39140cd460 con 0x7f39140ce2b0 2024-09-13T16:44:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.353+0000 7f391b200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39140ce810 con 0x7f39140cae30 2024-09-13T16:44:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.353+0000 7f3913400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39140ce2b0 0x7f39140b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.353+0000 7f3913400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39140ce2b0 0x7f39140b3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60956/0 (socket says 172.21.15.28:60956) 2024-09-13T16:44:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.353+0000 7f3913400640 1 -- 172.21.15.28:0/734125251 learned_addr learned my addr 172.21.15.28:0/734125251 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.354+0000 7f3919200640 1 --2- 172.21.15.28:0/734125251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39141f6190 0x7f39141f6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.354+0000 7f3913400640 1 -- 172.21.15.28:0/734125251 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39140cae30 msgr2=0x7f39140b2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.354+0000 7f3913400640 1 --2- 172.21.15.28:0/734125251 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39140cae30 0x7f39140b2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.354+0000 7f3913400640 1 -- 172.21.15.28:0/734125251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39141f6190 msgr2=0x7f39141f6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.354+0000 7f3913400640 1 --2- 172.21.15.28:0/734125251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39141f6190 0x7f39141f6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.354+0000 7f3913400640 1 -- 172.21.15.28:0/734125251 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39140ceaf0 con 0x7f39140ce2b0 2024-09-13T16:44:05.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.354+0000 7f3919200640 1 --2- 172.21.15.28:0/734125251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39141f6190 0x7f39141f6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:44:05.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.355+0000 7f3913400640 1 --2- 172.21.15.28:0/734125251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39140ce2b0 0x7f39140b3270 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f390002ac90 tx=0x7f39140b2700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.356+0000 7f390be00640 1 -- 172.21.15.28:0/734125251 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3900005a80 con 0x7f39140ce2b0 2024-09-13T16:44:05.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.356+0000 7f390be00640 1 -- 172.21.15.28:0/734125251 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3900005be0 con 0x7f39140ce2b0 2024-09-13T16:44:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.356+0000 7f390be00640 1 -- 172.21.15.28:0/734125251 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3900007070 con 0x7f39140ce2b0 2024-09-13T16:44:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.357+0000 7f391b200640 1 -- 172.21.15.28:0/734125251 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39140cedd0 con 0x7f39140ce2b0 2024-09-13T16:44:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.357+0000 7f391b200640 1 -- 172.21.15.28:0/734125251 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39140cf390 con 0x7f39140ce2b0 2024-09-13T16:44:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.358+0000 7f391b200640 1 -- 172.21.15.28:0/734125251 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3914067460 con 0x7f39140ce2b0 2024-09-13T16:44:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.362+0000 7f390be00640 1 -- 172.21.15.28:0/734125251 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f390002f030 con 0x7f39140ce2b0 2024-09-13T16:44:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.363+0000 7f390be00640 1 --2- 172.21.15.28:0/734125251 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f38f406ac00 0x7f38f406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.364+0000 7f390be00640 1 -- 172.21.15.28:0/734125251 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39000a9260 con 0x7f39140ce2b0 2024-09-13T16:44:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.364+0000 7f3913e00640 1 --2- 172.21.15.28:0/734125251 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f38f406ac00 0x7f38f406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.366+0000 7f3913e00640 1 --2- 172.21.15.28:0/734125251 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f38f406ac00 0x7f38f406d0c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3904006fd0 tx=0x7f3904008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.366+0000 7f390be00640 1 -- 172.21.15.28:0/734125251 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f39000744b0 con 0x7f39140ce2b0 2024-09-13T16:44:05.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.652+0000 7f98a7200640 1 -- 172.21.15.28:0/944102191 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f98a81eda50 con 0x7f98a8152e50 2024-09-13T16:44:05.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.653+0000 7f989e000640 1 -- 172.21.15.28:0/944102191 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f98a0018070 con 0x7f98a8152e50 2024-09-13T16:44:05.654 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-13T16:44:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.656+0000 7f98a7200640 1 -- 172.21.15.28:0/944102191 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f988806ac30 msgr2=0x7f988806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.656+0000 7f98a7200640 1 --2- 172.21.15.28:0/944102191 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f988806ac30 0x7f988806d0f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f989403d000 tx=0x7f9894008030 comp rx=0 tx=0).stop 2024-09-13T16:44:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.657+0000 7f98a7200640 1 -- 172.21.15.28:0/944102191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98a8152e50 msgr2=0x7f98a81553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.657+0000 7f98a7200640 1 --2- 172.21.15.28:0/944102191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98a8152e50 0x7f98a81553d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f98a000efc0 tx=0x7f98a000c490 comp rx=0 tx=0).stop 2024-09-13T16:44:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.657+0000 7f98a7200640 1 -- 172.21.15.28:0/944102191 shutdown_connections 2024-09-13T16:44:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.657+0000 7f98a7200640 1 --2- 172.21.15.28:0/944102191 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f988806ac30 0x7f988806d0f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.657+0000 7f98a7200640 1 --2- 172.21.15.28:0/944102191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98a8152e50 0x7f98a81553d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.657+0000 7f98a7200640 1 --2- 172.21.15.28:0/944102191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a80ca420 0x7f98a80b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.657+0000 7f98a7200640 1 --2- 172.21.15.28:0/944102191 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98a80c9230 0x7f98a81edda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.657+0000 7f98a7200640 1 -- 172.21.15.28:0/944102191 >> 172.21.15.28:0/944102191 conn(0x7f98a8065c50 msgr2=0x7f98a8150e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.657+0000 7f98a7200640 1 -- 172.21.15.28:0/944102191 shutdown_connections 2024-09-13T16:44:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.657+0000 7f98a7200640 1 -- 172.21.15.28:0/944102191 wait complete. 2024-09-13T16:44:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.668+0000 7f670f200640 1 -- 172.21.15.28:0/2235657249 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f67100c49d0 con 0x7f671006bfc0 2024-09-13T16:44:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.668+0000 7f6706000640 1 -- 172.21.15.28:0/2235657249 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6700017020 con 0x7f671006bfc0 2024-09-13T16:44:05.671 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-13T16:44:05.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.672+0000 7f3852600640 1 -- 172.21.15.28:0/1680971245 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f3828005500 con 0x7f3844044b00 2024-09-13T16:44:05.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.673+0000 7f670f200640 1 -- 172.21.15.28:0/2235657249 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f66f006ac30 msgr2=0x7f66f006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.673+0000 7f670f200640 1 --2- 172.21.15.28:0/2235657249 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f66f006ac30 0x7f66f006d0f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6708034000 tx=0x7f670802f000 comp rx=0 tx=0).stop 2024-09-13T16:44:05.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.673+0000 7f670f200640 1 -- 172.21.15.28:0/2235657249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f671006bfc0 msgr2=0x7f67100c3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.673+0000 7f670f200640 1 --2- 172.21.15.28:0/2235657249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f671006bfc0 0x7f67100c3bd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f670000c410 tx=0x7f670000c8d0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.673+0000 7f3848c00640 1 -- 172.21.15.28:0/1680971245 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f384c087020 con 0x7f3844044b00 2024-09-13T16:44:05.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.674+0000 7f670f200640 1 -- 172.21.15.28:0/2235657249 shutdown_connections 2024-09-13T16:44:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.674+0000 7f670f200640 1 --2- 172.21.15.28:0/2235657249 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f66f006ac30 0x7f66f006d0f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.674+0000 7f670f200640 1 --2- 172.21.15.28:0/2235657249 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6710156b80 0x7f6710158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.674+0000 7f670f200640 1 --2- 172.21.15.28:0/2235657249 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f671006bfc0 0x7f67100c3bd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.674+0000 7f670f200640 1 --2- 172.21.15.28:0/2235657249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f671006bbe0 0x7f67101f28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.674+0000 7f670f200640 1 -- 172.21.15.28:0/2235657249 >> 172.21.15.28:0/2235657249 conn(0x7f67100ca2f0 msgr2=0x7f67100c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.675+0000 7f670f200640 1 -- 172.21.15.28:0/2235657249 shutdown_connections 2024-09-13T16:44:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.675+0000 7f670f200640 1 -- 172.21.15.28:0/2235657249 wait complete. 2024-09-13T16:44:05.680 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-13T16:44:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.677+0000 7f3852600640 1 -- 172.21.15.28:0/1680971245 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f382c06ad00 msgr2=0x7f382c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.677+0000 7f3852600640 1 --2- 172.21.15.28:0/1680971245 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f382c06ad00 0x7f382c06d1c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3838034000 tx=0x7f383802e000 comp rx=0 tx=0).stop 2024-09-13T16:44:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.677+0000 7f3852600640 1 -- 172.21.15.28:0/1680971245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3844044b00 msgr2=0x7f384419e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.677+0000 7f3852600640 1 --2- 172.21.15.28:0/1680971245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3844044b00 0x7f384419e380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f384c074920 tx=0x7f384c067a20 comp rx=0 tx=0).stop 2024-09-13T16:44:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.678+0000 7f3852600640 1 -- 172.21.15.28:0/1680971245 shutdown_connections 2024-09-13T16:44:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.678+0000 7f3852600640 1 --2- 172.21.15.28:0/1680971245 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f382c06ad00 0x7f382c06d1c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.678+0000 7f3852600640 1 --2- 172.21.15.28:0/1680971245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f384404aa10 0x7f38440f3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.678+0000 7f3852600640 1 --2- 172.21.15.28:0/1680971245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3844044ee0 0x7f38440f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.678+0000 7f3852600640 1 --2- 172.21.15.28:0/1680971245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3844044b00 0x7f384419e380 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.679+0000 7f3852600640 1 -- 172.21.15.28:0/1680971245 >> 172.21.15.28:0/1680971245 conn(0x7f384404fc20 msgr2=0x7f384404db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.679+0000 7f3852600640 1 -- 172.21.15.28:0/1680971245 shutdown_connections 2024-09-13T16:44:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.679+0000 7f3852600640 1 -- 172.21.15.28:0/1680971245 wait complete. 2024-09-13T16:44:05.686 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.6 2024-09-13T16:44:05.686 DEBUG:teuthology.parallel:result is None 2024-09-13T16:44:05.704 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.2 2024-09-13T16:44:05.704 DEBUG:teuthology.parallel:result is None 2024-09-13T16:44:05.714 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.0 2024-09-13T16:44:05.714 DEBUG:teuthology.parallel:result is None 2024-09-13T16:44:05.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.722+0000 7f744e200640 1 -- 172.21.15.28:0/403522969 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7420005500 con 0x7f74480b3020 2024-09-13T16:44:05.725 INFO:teuthology.orchestra.run.smithi028.stdout:55834574853 2024-09-13T16:44:05.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.723+0000 7f743f400640 1 -- 172.21.15.28:0/403522969 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7438057d70 con 0x7f74480b3020 2024-09-13T16:44:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.725+0000 7f744e200640 1 -- 172.21.15.28:0/403522969 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f742806ac80 msgr2=0x7f742806d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.725+0000 7f744e200640 1 --2- 172.21.15.28:0/403522969 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f742806ac80 0x7f742806d140 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7440000f80 tx=0x7f7440009000 comp rx=0 tx=0).stop 2024-09-13T16:44:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.725+0000 7f744e200640 1 -- 172.21.15.28:0/403522969 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74480b3020 msgr2=0x7f74481e4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.725+0000 7f744e200640 1 --2- 172.21.15.28:0/403522969 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74480b3020 0x7f74481e4ee0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f743800e900 tx=0x7f743800edc0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.725+0000 7f744e200640 1 -- 172.21.15.28:0/403522969 shutdown_connections 2024-09-13T16:44:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.725+0000 7f744e200640 1 --2- 172.21.15.28:0/403522969 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f742806ac80 0x7f742806d140 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.725+0000 7f744e200640 1 --2- 172.21.15.28:0/403522969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74480ca260 0x7f74481e5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.725+0000 7f744e200640 1 --2- 172.21.15.28:0/403522969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74480b3400 0x7f74481520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.725+0000 7f744e200640 1 --2- 172.21.15.28:0/403522969 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74480b3020 0x7f74481e4ee0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.725+0000 7f744e200640 1 -- 172.21.15.28:0/403522969 >> 172.21.15.28:0/403522969 conn(0x7f74480666a0 msgr2=0x7f74480cc910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.725+0000 7f744e200640 1 -- 172.21.15.28:0/403522969 shutdown_connections 2024-09-13T16:44:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.725+0000 7f744e200640 1 -- 172.21.15.28:0/403522969 wait complete. 2024-09-13T16:44:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.728+0000 7f391b200640 1 -- 172.21.15.28:0/734125251 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f391415ad50 con 0x7f39140ce2b0 2024-09-13T16:44:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.729+0000 7f390be00640 1 -- 172.21.15.28:0/734125251 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3900041020 con 0x7f39140ce2b0 2024-09-13T16:44:05.730 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-13T16:44:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.731+0000 7f391b200640 1 -- 172.21.15.28:0/734125251 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f38f406ac00 msgr2=0x7f38f406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.731+0000 7f391b200640 1 --2- 172.21.15.28:0/734125251 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f38f406ac00 0x7f38f406d0c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3904006fd0 tx=0x7f3904008040 comp rx=0 tx=0).stop 2024-09-13T16:44:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.731+0000 7f391b200640 1 -- 172.21.15.28:0/734125251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39140ce2b0 msgr2=0x7f39140b3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.731+0000 7f391b200640 1 --2- 172.21.15.28:0/734125251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39140ce2b0 0x7f39140b3270 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f390002ac90 tx=0x7f39140b2700 comp rx=0 tx=0).stop 2024-09-13T16:44:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.731+0000 7f391b200640 1 -- 172.21.15.28:0/734125251 shutdown_connections 2024-09-13T16:44:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.731+0000 7f391b200640 1 --2- 172.21.15.28:0/734125251 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f38f406ac00 0x7f38f406d0c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.731+0000 7f391b200640 1 --2- 172.21.15.28:0/734125251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39141f6190 0x7f39141f6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.731+0000 7f391b200640 1 --2- 172.21.15.28:0/734125251 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39140ce2b0 0x7f39140b3270 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.731+0000 7f391b200640 1 --2- 172.21.15.28:0/734125251 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39140cae30 0x7f39140b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.731+0000 7f391b200640 1 -- 172.21.15.28:0/734125251 >> 172.21.15.28:0/734125251 conn(0x7f3914071790 msgr2=0x7f391414eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:05.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.731+0000 7f391b200640 1 -- 172.21.15.28:0/734125251 shutdown_connections 2024-09-13T16:44:05.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.731+0000 7f391b200640 1 -- 172.21.15.28:0/734125251 wait complete. 2024-09-13T16:44:05.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.739+0000 7fb207e00640 1 -- 172.21.15.28:0/1273436901 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fb200036df0 con 0x7fb2000dee10 2024-09-13T16:44:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.740+0000 7fb1fea00640 1 -- 172.21.15.28:0/1273436901 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb2080d1d40 con 0x7fb2000dee10 2024-09-13T16:44:05.741 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-13T16:44:05.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.743+0000 7fb207e00640 1 -- 172.21.15.28:0/1273436901 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb1e806ac00 msgr2=0x7fb1e806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.743+0000 7fb207e00640 1 --2- 172.21.15.28:0/1273436901 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb1e806ac00 0x7fb1e806d0c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8034000 tx=0x7fb1f802e000 comp rx=0 tx=0).stop 2024-09-13T16:44:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.743+0000 7fb207e00640 1 -- 172.21.15.28:0/1273436901 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2000dee10 msgr2=0x7fb200052560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.743+0000 7fb207e00640 1 --2- 172.21.15.28:0/1273436901 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2000dee10 0x7fb200052560 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb20804e4e0 tx=0x7fb208067310 comp rx=0 tx=0).stop 2024-09-13T16:44:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.743+0000 7fb207e00640 1 -- 172.21.15.28:0/1273436901 shutdown_connections 2024-09-13T16:44:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.743+0000 7fb207e00640 1 --2- 172.21.15.28:0/1273436901 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb1e806ac00 0x7fb1e806d0c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.743+0000 7fb207e00640 1 --2- 172.21.15.28:0/1273436901 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb2000dee10 0x7fb200052560 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.743+0000 7fb207e00640 1 --2- 172.21.15.28:0/1273436901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2000d8840 0x7fb2000ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.743+0000 7fb207e00640 1 --2- 172.21.15.28:0/1273436901 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2000502a0 0x7fb2000db570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.743+0000 7fb207e00640 1 -- 172.21.15.28:0/1273436901 >> 172.21.15.28:0/1273436901 conn(0x7fb200054420 msgr2=0x7fb20004e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.743+0000 7fb207e00640 1 -- 172.21.15.28:0/1273436901 shutdown_connections 2024-09-13T16:44:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.743+0000 7fb207e00640 1 -- 172.21.15.28:0/1273436901 wait complete. 2024-09-13T16:44:05.749 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2024-09-13T16:44:05.749 DEBUG:teuthology.parallel:result is None 2024-09-13T16:44:05.751 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.7 2024-09-13T16:44:05.751 DEBUG:teuthology.parallel:result is None 2024-09-13T16:44:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.756+0000 7f4746200640 1 -- 172.21.15.28:0/3261140187 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4740153480 con 0x7f4740157110 2024-09-13T16:44:05.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.757+0000 7f4737400640 1 -- 172.21.15.28:0/3261140187 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4740153480 con 0x7f4740157110 2024-09-13T16:44:05.758 INFO:teuthology.orchestra.run.smithi028.stdout:55834574853 2024-09-13T16:44:05.760 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.5 2024-09-13T16:44:05.761 DEBUG:teuthology.parallel:result is None 2024-09-13T16:44:05.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.761+0000 7f4746200640 1 -- 172.21.15.28:0/3261140187 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f472006ad00 msgr2=0x7f472006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.761+0000 7f4746200640 1 --2- 172.21.15.28:0/3261140187 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f472006ad00 0x7f472006d1c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4730034000 tx=0x7f473002e000 comp rx=0 tx=0).stop 2024-09-13T16:44:05.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.761+0000 7f4746200640 1 -- 172.21.15.28:0/3261140187 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4740157110 msgr2=0x7f474015e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.761+0000 7f4746200640 1 --2- 172.21.15.28:0/3261140187 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4740157110 0x7f474015e870 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f473800c890 tx=0x7f473800cd50 comp rx=0 tx=0).stop 2024-09-13T16:44:05.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.761+0000 7f4746200640 1 -- 172.21.15.28:0/3261140187 shutdown_connections 2024-09-13T16:44:05.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.761+0000 7f4746200640 1 --2- 172.21.15.28:0/3261140187 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f472006ad00 0x7f472006d1c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.761+0000 7f4746200640 1 --2- 172.21.15.28:0/3261140187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47401574f0 0x7f47401fabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.761+0000 7f4746200640 1 --2- 172.21.15.28:0/3261140187 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4740157110 0x7f474015e870 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.761+0000 7f4746200640 1 --2- 172.21.15.28:0/3261140187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f474006df90 0x7f474015e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.761+0000 7f4746200640 1 -- 172.21.15.28:0/3261140187 >> 172.21.15.28:0/3261140187 conn(0x7f47400ca6b0 msgr2=0x7f47400c24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:05.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.761+0000 7f4746200640 1 -- 172.21.15.28:0/3261140187 shutdown_connections 2024-09-13T16:44:05.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.761+0000 7f4746200640 1 -- 172.21.15.28:0/3261140187 wait complete. 2024-09-13T16:44:05.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.763+0000 7f132de00640 1 -- 172.21.15.28:0/990304764 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1328150560 con 0x7f13280c9b70 2024-09-13T16:44:05.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.764+0000 7f131f400640 1 -- 172.21.15.28:0/990304764 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1328150560 con 0x7f13280c9b70 2024-09-13T16:44:05.765 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-13T16:44:05.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.767+0000 7f132de00640 1 -- 172.21.15.28:0/990304764 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f130806ad00 msgr2=0x7f130806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.767+0000 7f132de00640 1 --2- 172.21.15.28:0/990304764 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f130806ad00 0x7f130806d1c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1318033000 tx=0x7f131802e000 comp rx=0 tx=0).stop 2024-09-13T16:44:05.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.768+0000 7f132de00640 1 -- 172.21.15.28:0/990304764 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13280c9b70 msgr2=0x7f13280c78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.768+0000 7f132de00640 1 --2- 172.21.15.28:0/990304764 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13280c9b70 0x7f13280c78c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f131400c480 tx=0x7f131400c940 comp rx=0 tx=0).stop 2024-09-13T16:44:05.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.768+0000 7f132de00640 1 -- 172.21.15.28:0/990304764 shutdown_connections 2024-09-13T16:44:05.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.768+0000 7f132de00640 1 --2- 172.21.15.28:0/990304764 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f130806ad00 0x7f130806d1c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.768+0000 7f132de00640 1 --2- 172.21.15.28:0/990304764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132815dd00 0x7f13280ce8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.768+0000 7f132de00640 1 --2- 172.21.15.28:0/990304764 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1328157310 0x7f13281511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.768+0000 7f132de00640 1 --2- 172.21.15.28:0/990304764 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13280c9b70 0x7f13280c78c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.768+0000 7f132de00640 1 -- 172.21.15.28:0/990304764 >> 172.21.15.28:0/990304764 conn(0x7f1328071790 msgr2=0x7f132815ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.768+0000 7f132de00640 1 -- 172.21.15.28:0/990304764 shutdown_connections 2024-09-13T16:44:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.768+0000 7f132de00640 1 -- 172.21.15.28:0/990304764 wait complete. 2024-09-13T16:44:05.784 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2024-09-13T16:44:05.784 DEBUG:teuthology.parallel:result is None 2024-09-13T16:44:05.787 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.3 2024-09-13T16:44:05.787 DEBUG:teuthology.parallel:result is None 2024-09-13T16:44:05.787 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T16:44:05.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:44:05.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.980+0000 7f4597a00640 1 Processor -- start 2024-09-13T16:44:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.980+0000 7f4597a00640 1 -- start start 2024-09-13T16:44:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.981+0000 7f4597a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4590150ca0 0x7f45901568e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.982+0000 7f4597a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f45900c4d60 0x7f45900c5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.982+0000 7f4597a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4590151080 con 0x7f45900c2250 2024-09-13T16:44:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.982+0000 7f4597a00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45900c2670 con 0x7f45900c4d60 2024-09-13T16:44:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.982+0000 7f4597a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459015ae40 con 0x7f4590150ca0 2024-09-13T16:44:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.982+0000 7f4587e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f45900c2250 0x7f459015a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59586/0 (socket says 172.21.15.28:59586) 2024-09-13T16:44:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.982+0000 7f4587e00640 1 -- 172.21.15.28:0/1196590503 learned_addr learned my addr 172.21.15.28:0/1196590503 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.982+0000 7f4595000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4590150ca0 0x7f45901568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.982+0000 7f4595a00640 1 --2- 172.21.15.28:0/1196590503 >> v2:172.21.15.191:3300/0 conn(0x7f45900c4d60 0x7f45900c5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.982+0000 7f4595000640 1 -- 172.21.15.28:0/1196590503 >> v2:172.21.15.191:3300/0 conn(0x7f45900c4d60 msgr2=0x7f45900c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.982+0000 7f4595000640 1 --2- 172.21.15.28:0/1196590503 >> v2:172.21.15.191:3300/0 conn(0x7f45900c4d60 0x7f45900c5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.982+0000 7f4595000640 1 -- 172.21.15.28:0/1196590503 >> v1:172.21.15.28:6789/0 conn(0x7f45900c2250 legacy=0x7f459015a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.982+0000 7f458fe00640 1 -- 172.21.15.28:0/1196590503 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881857381 0 0) 0x7f4590151080 con 0x7f45900c2250 2024-09-13T16:44:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.982+0000 7f4595000640 1 -- 172.21.15.28:0/1196590503 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459006a990 con 0x7f4590150ca0 2024-09-13T16:44:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.983+0000 7f4595000640 1 --2- 172.21.15.28:0/1196590503 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4590150ca0 0x7f45901568e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f45800097a0 tx=0x7f458002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c67a6ac20408018 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.984+0000 7f458fe00640 1 -- 172.21.15.28:0/1196590503 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4580032440 con 0x7f4590150ca0 2024-09-13T16:44:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.984+0000 7f458fe00640 1 -- 172.21.15.28:0/1196590503 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45800325a0 con 0x7f4590150ca0 2024-09-13T16:44:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.984+0000 7f458fe00640 1 -- 172.21.15.28:0/1196590503 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4580032930 con 0x7f4590150ca0 2024-09-13T16:44:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.984+0000 7f4597a00640 1 -- 172.21.15.28:0/1196590503 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4590150ca0 msgr2=0x7f45901568e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.984+0000 7f4597a00640 1 --2- 172.21.15.28:0/1196590503 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4590150ca0 0x7f45901568e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f45800097a0 tx=0x7f458002a180 comp rx=0 tx=0).stop 2024-09-13T16:44:05.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.984+0000 7f4597a00640 1 -- 172.21.15.28:0/1196590503 shutdown_connections 2024-09-13T16:44:05.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.984+0000 7f4597a00640 1 --2- 172.21.15.28:0/1196590503 >> v2:172.21.15.191:3300/0 conn(0x7f45900c4d60 0x7f45900c5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.984+0000 7f4597a00640 1 --2- 172.21.15.28:0/1196590503 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4590150ca0 0x7f45901568e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.984+0000 7f4597a00640 1 -- 172.21.15.28:0/1196590503 >> 172.21.15.28:0/1196590503 conn(0x7f4590066040 msgr2=0x7f4590069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:05.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.984+0000 7f4597a00640 1 -- 172.21.15.28:0/1196590503 shutdown_connections 2024-09-13T16:44:05.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.984+0000 7f4597a00640 1 -- 172.21.15.28:0/1196590503 wait complete. 2024-09-13T16:44:05.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.986+0000 7f4597a00640 1 Processor -- start 2024-09-13T16:44:05.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.986+0000 7f4597a00640 1 -- start start 2024-09-13T16:44:05.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.987+0000 7f4597a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45900c2250 0x7f4590158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.987+0000 7f4597a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45900c4d60 0x7f45901f0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.987+0000 7f4597a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4590150ca0 0x7f45901f0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.987+0000 7f4597a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4590148020 con 0x7f45900c2250 2024-09-13T16:44:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.987+0000 7f4597a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45901f0190 con 0x7f4590150ca0 2024-09-13T16:44:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.987+0000 7f4597a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4590148ba0 con 0x7f45900c4d60 2024-09-13T16:44:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.987+0000 7f4595000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45900c2250 0x7f4590158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.987+0000 7f4595000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45900c2250 0x7f4590158d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41792/0 (socket says 172.21.15.28:41792) 2024-09-13T16:44:05.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.987+0000 7f4595000640 1 -- 172.21.15.28:0/93675887 learned_addr learned my addr 172.21.15.28:0/93675887 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:05.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.987+0000 7f4595a00640 1 --2- 172.21.15.28:0/93675887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4590150ca0 0x7f45901f0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.987+0000 7f4587e00640 1 --2- 172.21.15.28:0/93675887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45900c4d60 0x7f45901f0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:05.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.987+0000 7f4595000640 1 -- 172.21.15.28:0/93675887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45900c4d60 msgr2=0x7f45901f0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.987+0000 7f4595000640 1 --2- 172.21.15.28:0/93675887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45900c4d60 0x7f45901f0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.987+0000 7f4595000640 1 -- 172.21.15.28:0/93675887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4590150ca0 msgr2=0x7f45901f0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:05.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.987+0000 7f4595000640 1 --2- 172.21.15.28:0/93675887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4590150ca0 0x7f45901f0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:05.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.987+0000 7f4595000640 1 -- 172.21.15.28:0/93675887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4580009400 con 0x7f45900c2250 2024-09-13T16:44:05.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.988+0000 7f4595000640 1 --2- 172.21.15.28:0/93675887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45900c2250 0x7f4590158d20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4580000c00 tx=0x7f45800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:05.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.988+0000 7f458d600640 1 -- 172.21.15.28:0/93675887 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4580030b80 con 0x7f45900c2250 2024-09-13T16:44:06.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.988+0000 7f458d600640 1 -- 172.21.15.28:0/93675887 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4580032e90 con 0x7f45900c2250 2024-09-13T16:44:06.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.988+0000 7f4597a00640 1 -- 172.21.15.28:0/93675887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45900701d0 con 0x7f45900c2250 2024-09-13T16:44:06.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.988+0000 7f458d600640 1 -- 172.21.15.28:0/93675887 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4580006e10 con 0x7f45900c2250 2024-09-13T16:44:06.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.988+0000 7f4597a00640 1 -- 172.21.15.28:0/93675887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459006fef0 con 0x7f45900c2250 2024-09-13T16:44:06.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.989+0000 7f4597a00640 1 -- 172.21.15.28:0/93675887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45900c36d0 con 0x7f45900c2250 2024-09-13T16:44:06.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.990+0000 7f458d600640 1 -- 172.21.15.28:0/93675887 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4580039040 con 0x7f45900c2250 2024-09-13T16:44:06.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.991+0000 7f458d600640 1 --2- 172.21.15.28:0/93675887 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f456c06ac00 0x7f456c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:06.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.991+0000 7f458d600640 1 -- 172.21.15.28:0/93675887 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45800a9cf0 con 0x7f45900c2250 2024-09-13T16:44:06.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.991+0000 7f4587e00640 1 --2- 172.21.15.28:0/93675887 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f456c06ac00 0x7f456c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:06.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.992+0000 7f4587e00640 1 --2- 172.21.15.28:0/93675887 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f456c06ac00 0x7f456c06d0c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f459014e140 tx=0x7f4578007d50 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:06.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:05.995+0000 7f458d600640 1 -- 172.21.15.28:0/93675887 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4580074ee0 con 0x7f45900c2250 2024-09-13T16:44:06.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.173+0000 7f4597a00640 1 -- 172.21.15.28:0/93675887 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f45900c2e30 con 0x7f456c06ac00 2024-09-13T16:44:06.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.180+0000 7f458d600640 1 -- 172.21.15.28:0/93675887 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264696 (secure 0 0 0) 0x7f4578004c70 con 0x7f456c06ac00 2024-09-13T16:44:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:06.189 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T16:44:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.184+0000 7f4597a00640 1 -- 172.21.15.28:0/93675887 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f456c06ac00 msgr2=0x7f456c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.184+0000 7f4597a00640 1 --2- 172.21.15.28:0/93675887 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f456c06ac00 0x7f456c06d0c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f459014e140 tx=0x7f4578007d50 comp rx=0 tx=0).stop 2024-09-13T16:44:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.184+0000 7f4597a00640 1 -- 172.21.15.28:0/93675887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45900c2250 msgr2=0x7f4590158d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.184+0000 7f4597a00640 1 --2- 172.21.15.28:0/93675887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45900c2250 0x7f4590158d20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4580000c00 tx=0x7f45800046c0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.185+0000 7f4597a00640 1 -- 172.21.15.28:0/93675887 shutdown_connections 2024-09-13T16:44:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.185+0000 7f4597a00640 1 --2- 172.21.15.28:0/93675887 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f456c06ac00 0x7f456c06d0c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.185+0000 7f4597a00640 1 --2- 172.21.15.28:0/93675887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4590150ca0 0x7f45901f0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.185+0000 7f4597a00640 1 --2- 172.21.15.28:0/93675887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45900c4d60 0x7f45901f0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.185+0000 7f4597a00640 1 --2- 172.21.15.28:0/93675887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45900c2250 0x7f4590158d20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.185+0000 7f4597a00640 1 -- 172.21.15.28:0/93675887 >> 172.21.15.28:0/93675887 conn(0x7f4590066040 msgr2=0x7f45901e04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:06.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.185+0000 7f4597a00640 1 -- 172.21.15.28:0/93675887 shutdown_connections 2024-09-13T16:44:06.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.185+0000 7f4597a00640 1 -- 172.21.15.28:0/93675887 wait complete. 2024-09-13T16:44:06.202 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-13T16:44:05.592829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222004,"kb_used_data":1780,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754752716,"statfs":{"total":773094113280,"available":772866781184,"internally_reserved":0,"allocated":1822720,"data_stored":1750397,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12743,"internal_metadata":225431097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002550"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178474+0000","last_change":"2024-09-13T16:43:54.987241+0000","last_active":"2024-09-13T16:43:59.178474+0000","last_peered":"2024-09-13T16:43:59.178474+0000","last_clean":"2024-09-13T16:43:59.178474+0000","last_became_active":"2024-09-13T16:43:54.986988+0000","last_became_peered":"2024-09-13T16:43:54.986988+0000","last_unstale":"2024-09-13T16:43:59.178474+0000","last_undegraded":"2024-09-13T16:43:59.178474+0000","last_fullsized":"2024-09-13T16:43:59.178474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975685+0000","last_change":"2024-09-13T16:43:55.001141+0000","last_active":"2024-09-13T16:43:55.975685+0000","last_peered":"2024-09-13T16:43:55.975685+0000","last_clean":"2024-09-13T16:43:55.975685+0000","last_became_active":"2024-09-13T16:43:55.000757+0000","last_became_peered":"2024-09-13T16:43:55.000757+0000","last_unstale":"2024-09-13T16:43:55.975685+0000","last_undegraded":"2024-09-13T16:43:55.975685+0000","last_fullsized":"2024-09-13T16:43:55.975685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872108+0000","last_change":"2024-09-13T16:43:54.986925+0000","last_active":"2024-09-13T16:43:56.872108+0000","last_peered":"2024-09-13T16:43:56.872108+0000","last_clean":"2024-09-13T16:43:56.872108+0000","last_became_active":"2024-09-13T16:43:54.986198+0000","last_became_peered":"2024-09-13T16:43:54.986198+0000","last_unstale":"2024-09-13T16:43:56.872108+0000","last_undegraded":"2024-09-13T16:43:56.872108+0000","last_fullsized":"2024-09-13T16:43:56.872108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:03.127550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485981+0000","last_change":"2024-09-13T16:43:54.992249+0000","last_active":"2024-09-13T16:43:55.485981+0000","last_peered":"2024-09-13T16:43:55.485981+0000","last_clean":"2024-09-13T16:43:55.485981+0000","last_became_active":"2024-09-13T16:43:54.991804+0000","last_became_peered":"2024-09-13T16:43:54.991804+0000","last_unstale":"2024-09-13T16:43:55.485981+0000","last_undegraded":"2024-09-13T16:43:55.485981+0000","last_fullsized":"2024-09-13T16:43:55.485981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178163+0000","last_change":"2024-09-13T16:43:54.987564+0000","last_active":"2024-09-13T16:43:59.178163+0000","last_peered":"2024-09-13T16:43:59.178163+0000","last_clean":"2024-09-13T16:43:59.178163+0000","last_became_active":"2024-09-13T16:43:54.987328+0000","last_became_peered":"2024-09-13T16:43:54.987328+0000","last_unstale":"2024-09-13T16:43:59.178163+0000","last_undegraded":"2024-09-13T16:43:59.178163+0000","last_fullsized":"2024-09-13T16:43:59.178163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:36.006730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485919+0000","last_change":"2024-09-13T16:43:55.000019+0000","last_active":"2024-09-13T16:43:55.485919+0000","last_peered":"2024-09-13T16:43:55.485919+0000","last_clean":"2024-09-13T16:43:55.485919+0000","last_became_active":"2024-09-13T16:43:54.999752+0000","last_became_peered":"2024-09-13T16:43:54.999752+0000","last_unstale":"2024-09-13T16:43:55.485919+0000","last_undegraded":"2024-09-13T16:43:55.485919+0000","last_fullsized":"2024-09-13T16:43:55.485919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.177985+0000","last_change":"2024-09-13T16:43:54.999533+0000","last_active":"2024-09-13T16:43:59.177985+0000","last_peered":"2024-09-13T16:43:59.177985+0000","last_clean":"2024-09-13T16:43:59.177985+0000","last_became_active":"2024-09-13T16:43:54.999350+0000","last_became_peered":"2024-09-13T16:43:54.999350+0000","last_unstale":"2024-09-13T16:43:59.177985+0000","last_undegraded":"2024-09-13T16:43:59.177985+0000","last_fullsized":"2024-09-13T16:43:59.177985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:29.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508531+0000","last_change":"2024-09-13T16:43:54.998526+0000","last_active":"2024-09-13T16:43:56.508531+0000","last_peered":"2024-09-13T16:43:56.508531+0000","last_clean":"2024-09-13T16:43:56.508531+0000","last_became_active":"2024-09-13T16:43:54.996309+0000","last_became_peered":"2024-09-13T16:43:54.996309+0000","last_unstale":"2024-09-13T16:43:56.508531+0000","last_undegraded":"2024-09-13T16:43:56.508531+0000","last_fullsized":"2024-09-13T16:43:56.508531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178473+0000","last_change":"2024-09-13T16:43:54.993786+0000","last_active":"2024-09-13T16:43:59.178473+0000","last_peered":"2024-09-13T16:43:59.178473+0000","last_clean":"2024-09-13T16:43:59.178473+0000","last_became_active":"2024-09-13T16:43:54.993475+0000","last_became_peered":"2024-09-13T16:43:54.993475+0000","last_unstale":"2024-09-13T16:43:59.178473+0000","last_undegraded":"2024-09-13T16:43:59.178473+0000","last_fullsized":"2024-09-13T16:43:59.178473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260988+0000","last_change":"2024-09-13T16:43:54.986272+0000","last_active":"2024-09-13T16:43:56.260988+0000","last_peered":"2024-09-13T16:43:56.260988+0000","last_clean":"2024-09-13T16:43:56.260988+0000","last_became_active":"2024-09-13T16:43:54.985979+0000","last_became_peered":"2024-09-13T16:43:54.985979+0000","last_unstale":"2024-09-13T16:43:56.260988+0000","last_undegraded":"2024-09-13T16:43:56.260988+0000","last_fullsized":"2024-09-13T16:43:56.260988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260766+0000","last_change":"2024-09-13T16:43:54.986207+0000","last_active":"2024-09-13T16:43:56.260766+0000","last_peered":"2024-09-13T16:43:56.260766+0000","last_clean":"2024-09-13T16:43:56.260766+0000","last_became_active":"2024-09-13T16:43:54.985948+0000","last_became_peered":"2024-09-13T16:43:54.985948+0000","last_unstale":"2024-09-13T16:43:56.260766+0000","last_undegraded":"2024-09-13T16:43:56.260766+0000","last_fullsized":"2024-09-13T16:43:56.260766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935591+0000","last_change":"2024-09-13T16:43:54.986636+0000","last_active":"2024-09-13T16:43:56.935591+0000","last_peered":"2024-09-13T16:43:56.935591+0000","last_clean":"2024-09-13T16:43:56.935591+0000","last_became_active":"2024-09-13T16:43:54.986161+0000","last_became_peered":"2024-09-13T16:43:54.986161+0000","last_unstale":"2024-09-13T16:43:56.935591+0000","last_undegraded":"2024-09-13T16:43:56.935591+0000","last_fullsized":"2024-09-13T16:43:56.935591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:09.340330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508720+0000","last_change":"2024-09-13T16:43:54.997783+0000","last_active":"2024-09-13T16:43:56.508720+0000","last_peered":"2024-09-13T16:43:56.508720+0000","last_clean":"2024-09-13T16:43:56.508720+0000","last_became_active":"2024-09-13T16:43:54.997162+0000","last_became_peered":"2024-09-13T16:43:54.997162+0000","last_unstale":"2024-09-13T16:43:56.508720+0000","last_undegraded":"2024-09-13T16:43:56.508720+0000","last_fullsized":"2024-09-13T16:43:56.508720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872216+0000","last_change":"2024-09-13T16:43:54.995262+0000","last_active":"2024-09-13T16:43:56.872216+0000","last_peered":"2024-09-13T16:43:56.872216+0000","last_clean":"2024-09-13T16:43:56.872216+0000","last_became_active":"2024-09-13T16:43:54.994891+0000","last_became_peered":"2024-09-13T16:43:54.994891+0000","last_unstale":"2024-09-13T16:43:56.872216+0000","last_undegraded":"2024-09-13T16:43:56.872216+0000","last_fullsized":"2024-09-13T16:43:56.872216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975286+0000","last_change":"2024-09-13T16:43:54.981534+0000","last_active":"2024-09-13T16:43:55.975286+0000","last_peered":"2024-09-13T16:43:55.975286+0000","last_clean":"2024-09-13T16:43:55.975286+0000","last_became_active":"2024-09-13T16:43:54.981191+0000","last_became_peered":"2024-09-13T16:43:54.981191+0000","last_unstale":"2024-09-13T16:43:55.975286+0000","last_undegraded":"2024-09-13T16:43:55.975286+0000","last_fullsized":"2024-09-13T16:43:55.975286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485880+0000","last_change":"2024-09-13T16:43:54.994157+0000","last_active":"2024-09-13T16:43:55.485880+0000","last_peered":"2024-09-13T16:43:55.485880+0000","last_clean":"2024-09-13T16:43:55.485880+0000","last_became_active":"2024-09-13T16:43:54.993915+0000","last_became_peered":"2024-09-13T16:43:54.993915+0000","last_unstale":"2024-09-13T16:43:55.485880+0000","last_undegraded":"2024-09-13T16:43:55.485880+0000","last_fullsized":"2024-09-13T16:43:55.485880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486053+0000","last_change":"2024-09-13T16:43:54.987328+0000","last_active":"2024-09-13T16:43:55.486053+0000","last_peered":"2024-09-13T16:43:55.486053+0000","last_clean":"2024-09-13T16:43:55.486053+0000","last_became_active":"2024-09-13T16:43:54.987048+0000","last_became_peered":"2024-09-13T16:43:54.987048+0000","last_unstale":"2024-09-13T16:43:55.486053+0000","last_undegraded":"2024-09-13T16:43:55.486053+0000","last_fullsized":"2024-09-13T16:43:55.486053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.979840+0000","last_change":"2024-09-13T16:43:54.985628+0000","last_active":"2024-09-13T16:43:55.979840+0000","last_peered":"2024-09-13T16:43:55.979840+0000","last_clean":"2024-09-13T16:43:55.979840+0000","last_became_active":"2024-09-13T16:43:54.985378+0000","last_became_peered":"2024-09-13T16:43:54.985378+0000","last_unstale":"2024-09-13T16:43:55.979840+0000","last_undegraded":"2024-09-13T16:43:55.979840+0000","last_fullsized":"2024-09-13T16:43:55.979840+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178469+0000","last_change":"2024-09-13T16:43:54.996712+0000","last_active":"2024-09-13T16:43:59.178469+0000","last_peered":"2024-09-13T16:43:59.178469+0000","last_clean":"2024-09-13T16:43:59.178469+0000","last_became_active":"2024-09-13T16:43:54.996523+0000","last_became_peered":"2024-09-13T16:43:54.996523+0000","last_unstale":"2024-09-13T16:43:59.178469+0000","last_undegraded":"2024-09-13T16:43:59.178469+0000","last_fullsized":"2024-09-13T16:43:59.178469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260867+0000","last_change":"2024-09-13T16:43:54.984879+0000","last_active":"2024-09-13T16:43:56.260867+0000","last_peered":"2024-09-13T16:43:56.260867+0000","last_clean":"2024-09-13T16:43:56.260867+0000","last_became_active":"2024-09-13T16:43:54.984595+0000","last_became_peered":"2024-09-13T16:43:54.984595+0000","last_unstale":"2024-09-13T16:43:56.260867+0000","last_undegraded":"2024-09-13T16:43:56.260867+0000","last_fullsized":"2024-09-13T16:43:56.260867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.871914+0000","last_change":"2024-09-13T16:43:54.996457+0000","last_active":"2024-09-13T16:43:56.871914+0000","last_peered":"2024-09-13T16:43:56.871914+0000","last_clean":"2024-09-13T16:43:56.871914+0000","last_became_active":"2024-09-13T16:43:54.996275+0000","last_became_peered":"2024-09-13T16:43:54.996275+0000","last_unstale":"2024-09-13T16:43:56.871914+0000","last_undegraded":"2024-09-13T16:43:56.871914+0000","last_fullsized":"2024-09-13T16:43:56.871914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485929+0000","last_change":"2024-09-13T16:43:54.985368+0000","last_active":"2024-09-13T16:43:55.485929+0000","last_peered":"2024-09-13T16:43:55.485929+0000","last_clean":"2024-09-13T16:43:55.485929+0000","last_became_active":"2024-09-13T16:43:54.985130+0000","last_became_peered":"2024-09-13T16:43:54.985130+0000","last_unstale":"2024-09-13T16:43:55.485929+0000","last_undegraded":"2024-09-13T16:43:55.485929+0000","last_fullsized":"2024-09-13T16:43:55.485929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508531+0000","last_change":"2024-09-13T16:43:54.991862+0000","last_active":"2024-09-13T16:43:56.508531+0000","last_peered":"2024-09-13T16:43:56.508531+0000","last_clean":"2024-09-13T16:43:56.508531+0000","last_became_active":"2024-09-13T16:43:54.991554+0000","last_became_peered":"2024-09-13T16:43:54.991554+0000","last_unstale":"2024-09-13T16:43:56.508531+0000","last_undegraded":"2024-09-13T16:43:56.508531+0000","last_fullsized":"2024-09-13T16:43:56.508531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485896+0000","last_change":"2024-09-13T16:43:54.992241+0000","last_active":"2024-09-13T16:43:55.485896+0000","last_peered":"2024-09-13T16:43:55.485896+0000","last_clean":"2024-09-13T16:43:55.485896+0000","last_became_active":"2024-09-13T16:43:54.991687+0000","last_became_peered":"2024-09-13T16:43:54.991687+0000","last_unstale":"2024-09-13T16:43:55.485896+0000","last_undegraded":"2024-09-13T16:43:55.485896+0000","last_fullsized":"2024-09-13T16:43:55.485896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.509024+0000","last_change":"2024-09-13T16:43:54.991366+0000","last_active":"2024-09-13T16:43:56.509024+0000","last_peered":"2024-09-13T16:43:56.509024+0000","last_clean":"2024-09-13T16:43:56.509024+0000","last_became_active":"2024-09-13T16:43:54.991127+0000","last_became_peered":"2024-09-13T16:43:54.991127+0000","last_unstale":"2024-09-13T16:43:56.509024+0000","last_undegraded":"2024-09-13T16:43:56.509024+0000","last_fullsized":"2024-09-13T16:43:56.509024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.261006+0000","last_change":"2024-09-13T16:43:54.985615+0000","last_active":"2024-09-13T16:43:56.261006+0000","last_peered":"2024-09-13T16:43:56.261006+0000","last_clean":"2024-09-13T16:43:56.261006+0000","last_became_active":"2024-09-13T16:43:54.985299+0000","last_became_peered":"2024-09-13T16:43:54.985299+0000","last_unstale":"2024-09-13T16:43:56.261006+0000","last_undegraded":"2024-09-13T16:43:56.261006+0000","last_fullsized":"2024-09-13T16:43:56.261006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975382+0000","last_change":"2024-09-13T16:43:54.993325+0000","last_active":"2024-09-13T16:43:55.975382+0000","last_peered":"2024-09-13T16:43:55.975382+0000","last_clean":"2024-09-13T16:43:55.975382+0000","last_became_active":"2024-09-13T16:43:54.993009+0000","last_became_peered":"2024-09-13T16:43:54.993009+0000","last_unstale":"2024-09-13T16:43:55.975382+0000","last_undegraded":"2024-09-13T16:43:55.975382+0000","last_fullsized":"2024-09-13T16:43:55.975382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508820+0000","last_change":"2024-09-13T16:43:54.998229+0000","last_active":"2024-09-13T16:43:56.508820+0000","last_peered":"2024-09-13T16:43:56.508820+0000","last_clean":"2024-09-13T16:43:56.508820+0000","last_became_active":"2024-09-13T16:43:54.995774+0000","last_became_peered":"2024-09-13T16:43:54.995774+0000","last_unstale":"2024-09-13T16:43:56.508820+0000","last_undegraded":"2024-09-13T16:43:56.508820+0000","last_fullsized":"2024-09-13T16:43:56.508820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872235+0000","last_change":"2024-09-13T16:43:55.002126+0000","last_active":"2024-09-13T16:43:56.872235+0000","last_peered":"2024-09-13T16:43:56.872235+0000","last_clean":"2024-09-13T16:43:56.872235+0000","last_became_active":"2024-09-13T16:43:55.001713+0000","last_became_peered":"2024-09-13T16:43:55.001713+0000","last_unstale":"2024-09-13T16:43:56.872235+0000","last_undegraded":"2024-09-13T16:43:56.872235+0000","last_fullsized":"2024-09-13T16:43:56.872235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485939+0000","last_change":"2024-09-13T16:43:54.992745+0000","last_active":"2024-09-13T16:43:55.485939+0000","last_peered":"2024-09-13T16:43:55.485939+0000","last_clean":"2024-09-13T16:43:55.485939+0000","last_became_active":"2024-09-13T16:43:54.992508+0000","last_became_peered":"2024-09-13T16:43:54.992508+0000","last_unstale":"2024-09-13T16:43:55.485939+0000","last_undegraded":"2024-09-13T16:43:55.485939+0000","last_fullsized":"2024-09-13T16:43:55.485939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:23.913470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.979793+0000","last_change":"2024-09-13T16:43:54.990631+0000","last_active":"2024-09-13T16:43:55.979793+0000","last_peered":"2024-09-13T16:43:55.979793+0000","last_clean":"2024-09-13T16:43:55.979793+0000","last_became_active":"2024-09-13T16:43:54.986277+0000","last_became_peered":"2024-09-13T16:43:54.986277+0000","last_unstale":"2024-09-13T16:43:55.979793+0000","last_undegraded":"2024-09-13T16:43:55.979793+0000","last_fullsized":"2024-09-13T16:43:55.979793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260674+0000","last_change":"2024-09-13T16:43:55.000059+0000","last_active":"2024-09-13T16:43:56.260674+0000","last_peered":"2024-09-13T16:43:56.260674+0000","last_clean":"2024-09-13T16:43:56.260674+0000","last_became_active":"2024-09-13T16:43:54.999778+0000","last_became_peered":"2024-09-13T16:43:54.999778+0000","last_unstale":"2024-09-13T16:43:56.260674+0000","last_undegraded":"2024-09-13T16:43:56.260674+0000","last_fullsized":"2024-09-13T16:43:56.260674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935551+0000","last_change":"2024-09-13T16:43:54.988375+0000","last_active":"2024-09-13T16:43:56.935551+0000","last_peered":"2024-09-13T16:43:56.935551+0000","last_clean":"2024-09-13T16:43:56.935551+0000","last_became_active":"2024-09-13T16:43:54.988131+0000","last_became_peered":"2024-09-13T16:43:54.988131+0000","last_unstale":"2024-09-13T16:43:56.935551+0000","last_undegraded":"2024-09-13T16:43:56.935551+0000","last_fullsized":"2024-09-13T16:43:56.935551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.635755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508811+0000","last_change":"2024-09-13T16:43:54.555042+0000","last_active":"2024-09-13T16:43:56.508811+0000","last_peered":"2024-09-13T16:43:56.508811+0000","last_clean":"2024-09-13T16:43:56.508811+0000","last_became_active":"2024-09-13T16:43:54.554697+0000","last_became_peered":"2024-09-13T16:43:54.554697+0000","last_unstale":"2024-09-13T16:43:56.508811+0000","last_undegraded":"2024-09-13T16:43:56.508811+0000","last_fullsized":"2024-09-13T16:43:56.508811+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872075+0000","last_change":"2024-09-13T16:43:54.998755+0000","last_active":"2024-09-13T16:43:56.872075+0000","last_peered":"2024-09-13T16:43:56.872075+0000","last_clean":"2024-09-13T16:43:56.872075+0000","last_became_active":"2024-09-13T16:43:54.998399+0000","last_became_peered":"2024-09-13T16:43:54.998399+0000","last_unstale":"2024-09-13T16:43:56.872075+0000","last_undegraded":"2024-09-13T16:43:56.872075+0000","last_fullsized":"2024-09-13T16:43:56.872075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260801+0000","last_change":"2024-09-13T16:43:54.553250+0000","last_active":"2024-09-13T16:43:56.260801+0000","last_peered":"2024-09-13T16:43:56.260801+0000","last_clean":"2024-09-13T16:43:56.260801+0000","last_became_active":"2024-09-13T16:43:54.552977+0000","last_became_peered":"2024-09-13T16:43:54.552977+0000","last_unstale":"2024-09-13T16:43:56.260801+0000","last_undegraded":"2024-09-13T16:43:56.260801+0000","last_fullsized":"2024-09-13T16:43:56.260801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.984357+0000","last_change":"2024-09-13T16:43:54.558065+0000","last_active":"2024-09-13T16:43:54.984357+0000","last_peered":"2024-09-13T16:43:54.984357+0000","last_clean":"2024-09-13T16:43:54.984357+0000","last_became_active":"2024-09-13T16:43:54.557722+0000","last_became_peered":"2024-09-13T16:43:54.557722+0000","last_unstale":"2024-09-13T16:43:54.984357+0000","last_undegraded":"2024-09-13T16:43:54.984357+0000","last_fullsized":"2024-09-13T16:43:54.984357+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508519+0000","last_change":"2024-09-13T16:43:55.002041+0000","last_active":"2024-09-13T16:43:56.508519+0000","last_peered":"2024-09-13T16:43:56.508519+0000","last_clean":"2024-09-13T16:43:56.508519+0000","last_became_active":"2024-09-13T16:43:55.001512+0000","last_became_peered":"2024-09-13T16:43:55.001512+0000","last_unstale":"2024-09-13T16:43:56.508519+0000","last_undegraded":"2024-09-13T16:43:56.508519+0000","last_fullsized":"2024-09-13T16:43:56.508519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485908+0000","last_change":"2024-09-13T16:43:54.550389+0000","last_active":"2024-09-13T16:43:55.485908+0000","last_peered":"2024-09-13T16:43:55.485908+0000","last_clean":"2024-09-13T16:43:55.485908+0000","last_became_active":"2024-09-13T16:43:54.550154+0000","last_became_peered":"2024-09-13T16:43:54.550154+0000","last_unstale":"2024-09-13T16:43:55.485908+0000","last_undegraded":"2024-09-13T16:43:55.485908+0000","last_fullsized":"2024-09-13T16:43:55.485908+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975641+0000","last_change":"2024-09-13T16:43:54.993377+0000","last_active":"2024-09-13T16:43:55.975641+0000","last_peered":"2024-09-13T16:43:55.975641+0000","last_clean":"2024-09-13T16:43:55.975641+0000","last_became_active":"2024-09-13T16:43:54.993036+0000","last_became_peered":"2024-09-13T16:43:54.993036+0000","last_unstale":"2024-09-13T16:43:55.975641+0000","last_undegraded":"2024-09-13T16:43:55.975641+0000","last_fullsized":"2024-09-13T16:43:55.975641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.978710+0000","last_change":"2024-09-13T16:43:54.555528+0000","last_active":"2024-09-13T16:43:54.978710+0000","last_peered":"2024-09-13T16:43:54.978710+0000","last_clean":"2024-09-13T16:43:54.978710+0000","last_became_active":"2024-09-13T16:43:54.555267+0000","last_became_peered":"2024-09-13T16:43:54.555267+0000","last_unstale":"2024-09-13T16:43:54.978710+0000","last_undegraded":"2024-09-13T16:43:54.978710+0000","last_fullsized":"2024-09-13T16:43:54.978710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486032+0000","last_change":"2024-09-13T16:43:54.981706+0000","last_active":"2024-09-13T16:43:55.486032+0000","last_peered":"2024-09-13T16:43:55.486032+0000","last_clean":"2024-09-13T16:43:55.486032+0000","last_became_active":"2024-09-13T16:43:54.981429+0000","last_became_peered":"2024-09-13T16:43:54.981429+0000","last_unstale":"2024-09-13T16:43:55.486032+0000","last_undegraded":"2024-09-13T16:43:55.486032+0000","last_fullsized":"2024-09-13T16:43:55.486032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486042+0000","last_change":"2024-09-13T16:43:54.554593+0000","last_active":"2024-09-13T16:43:55.486042+0000","last_peered":"2024-09-13T16:43:55.486042+0000","last_clean":"2024-09-13T16:43:55.486042+0000","last_became_active":"2024-09-13T16:43:54.554307+0000","last_became_peered":"2024-09-13T16:43:54.554307+0000","last_unstale":"2024-09-13T16:43:55.486042+0000","last_undegraded":"2024-09-13T16:43:55.486042+0000","last_fullsized":"2024-09-13T16:43:55.486042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872139+0000","last_change":"2024-09-13T16:43:54.993485+0000","last_active":"2024-09-13T16:43:56.872139+0000","last_peered":"2024-09-13T16:43:56.872139+0000","last_clean":"2024-09-13T16:43:56.872139+0000","last_became_active":"2024-09-13T16:43:54.993226+0000","last_became_peered":"2024-09-13T16:43:54.993226+0000","last_unstale":"2024-09-13T16:43:56.872139+0000","last_undegraded":"2024-09-13T16:43:56.872139+0000","last_fullsized":"2024-09-13T16:43:56.872139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.982414+0000","last_change":"2024-09-13T16:43:54.562976+0000","last_active":"2024-09-13T16:43:59.982414+0000","last_peered":"2024-09-13T16:43:59.982414+0000","last_clean":"2024-09-13T16:43:59.982414+0000","last_became_active":"2024-09-13T16:43:54.560480+0000","last_became_peered":"2024-09-13T16:43:54.560480+0000","last_unstale":"2024-09-13T16:43:59.982414+0000","last_undegraded":"2024-09-13T16:43:59.982414+0000","last_fullsized":"2024-09-13T16:43:59.982414+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935689+0000","last_change":"2024-09-13T16:43:54.986459+0000","last_active":"2024-09-13T16:43:56.935689+0000","last_peered":"2024-09-13T16:43:56.935689+0000","last_clean":"2024-09-13T16:43:56.935689+0000","last_became_active":"2024-09-13T16:43:54.986083+0000","last_became_peered":"2024-09-13T16:43:54.986083+0000","last_unstale":"2024-09-13T16:43:56.935689+0000","last_undegraded":"2024-09-13T16:43:56.935689+0000","last_fullsized":"2024-09-13T16:43:56.935689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:51.960459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.985320+0000","last_change":"2024-09-13T16:43:54.565929+0000","last_active":"2024-09-13T16:43:59.985320+0000","last_peered":"2024-09-13T16:43:59.985320+0000","last_clean":"2024-09-13T16:43:59.985320+0000","last_became_active":"2024-09-13T16:43:54.565701+0000","last_became_peered":"2024-09-13T16:43:54.565701+0000","last_unstale":"2024-09-13T16:43:59.985320+0000","last_undegraded":"2024-09-13T16:43:59.985320+0000","last_fullsized":"2024-09-13T16:43:59.985320+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178232+0000","last_change":"2024-09-13T16:43:54.989068+0000","last_active":"2024-09-13T16:43:59.178232+0000","last_peered":"2024-09-13T16:43:59.178232+0000","last_clean":"2024-09-13T16:43:59.178232+0000","last_became_active":"2024-09-13T16:43:54.988779+0000","last_became_peered":"2024-09-13T16:43:54.988779+0000","last_unstale":"2024-09-13T16:43:59.178232+0000","last_undegraded":"2024-09-13T16:43:59.178232+0000","last_fullsized":"2024-09-13T16:43:59.178232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.982285+0000","last_change":"2024-09-13T16:43:54.561487+0000","last_active":"2024-09-13T16:43:59.982285+0000","last_peered":"2024-09-13T16:43:59.982285+0000","last_clean":"2024-09-13T16:43:59.982285+0000","last_became_active":"2024-09-13T16:43:54.561077+0000","last_became_peered":"2024-09-13T16:43:54.561077+0000","last_unstale":"2024-09-13T16:43:59.982285+0000","last_undegraded":"2024-09-13T16:43:59.982285+0000","last_fullsized":"2024-09-13T16:43:59.982285+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872141+0000","last_change":"2024-09-13T16:43:54.999517+0000","last_active":"2024-09-13T16:43:56.872141+0000","last_peered":"2024-09-13T16:43:56.872141+0000","last_clean":"2024-09-13T16:43:56.872141+0000","last_became_active":"2024-09-13T16:43:54.999283+0000","last_became_peered":"2024-09-13T16:43:54.999283+0000","last_unstale":"2024-09-13T16:43:56.872141+0000","last_undegraded":"2024-09-13T16:43:56.872141+0000","last_fullsized":"2024-09-13T16:43:56.872141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:20.207775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485949+0000","last_change":"2024-09-13T16:43:54.542871+0000","last_active":"2024-09-13T16:43:55.485949+0000","last_peered":"2024-09-13T16:43:55.485949+0000","last_clean":"2024-09-13T16:43:55.485949+0000","last_became_active":"2024-09-13T16:43:54.542548+0000","last_became_peered":"2024-09-13T16:43:54.542548+0000","last_unstale":"2024-09-13T16:43:55.485949+0000","last_undegraded":"2024-09-13T16:43:55.485949+0000","last_fullsized":"2024-09-13T16:43:55.485949+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178181+0000","last_change":"2024-09-13T16:43:54.989168+0000","last_active":"2024-09-13T16:43:59.178181+0000","last_peered":"2024-09-13T16:43:59.178181+0000","last_clean":"2024-09-13T16:43:59.178181+0000","last_became_active":"2024-09-13T16:43:54.988716+0000","last_became_peered":"2024-09-13T16:43:54.988716+0000","last_unstale":"2024-09-13T16:43:59.178181+0000","last_undegraded":"2024-09-13T16:43:59.178181+0000","last_fullsized":"2024-09-13T16:43:59.178181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.978600+0000","last_change":"2024-09-13T16:43:54.551551+0000","last_active":"2024-09-13T16:43:54.978600+0000","last_peered":"2024-09-13T16:43:54.978600+0000","last_clean":"2024-09-13T16:43:54.978600+0000","last_became_active":"2024-09-13T16:43:54.551293+0000","last_became_peered":"2024-09-13T16:43:54.551293+0000","last_unstale":"2024-09-13T16:43:54.978600+0000","last_undegraded":"2024-09-13T16:43:54.978600+0000","last_fullsized":"2024-09-13T16:43:54.978600+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260677+0000","last_change":"2024-09-13T16:43:54.994516+0000","last_active":"2024-09-13T16:43:56.260677+0000","last_peered":"2024-09-13T16:43:56.260677+0000","last_clean":"2024-09-13T16:43:56.260677+0000","last_became_active":"2024-09-13T16:43:54.994245+0000","last_became_peered":"2024-09-13T16:43:54.994245+0000","last_unstale":"2024-09-13T16:43:56.260677+0000","last_undegraded":"2024-09-13T16:43:56.260677+0000","last_fullsized":"2024-09-13T16:43:56.260677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.985709+0000","last_change":"2024-09-13T16:43:54.562232+0000","last_active":"2024-09-13T16:43:59.985709+0000","last_peered":"2024-09-13T16:43:59.985709+0000","last_clean":"2024-09-13T16:43:59.985709+0000","last_became_active":"2024-09-13T16:43:54.561939+0000","last_became_peered":"2024-09-13T16:43:54.561939+0000","last_unstale":"2024-09-13T16:43:59.985709+0000","last_undegraded":"2024-09-13T16:43:59.985709+0000","last_fullsized":"2024-09-13T16:43:59.985709+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178047+0000","last_change":"2024-09-13T16:43:54.988439+0000","last_active":"2024-09-13T16:43:59.178047+0000","last_peered":"2024-09-13T16:43:59.178047+0000","last_clean":"2024-09-13T16:43:59.178047+0000","last_became_active":"2024-09-13T16:43:54.988172+0000","last_became_peered":"2024-09-13T16:43:54.988172+0000","last_unstale":"2024-09-13T16:43:59.178047+0000","last_undegraded":"2024-09-13T16:43:59.178047+0000","last_fullsized":"2024-09-13T16:43:59.178047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935690+0000","last_change":"2024-09-13T16:43:54.559372+0000","last_active":"2024-09-13T16:43:56.935690+0000","last_peered":"2024-09-13T16:43:56.935690+0000","last_clean":"2024-09-13T16:43:56.935690+0000","last_became_active":"2024-09-13T16:43:54.559111+0000","last_became_peered":"2024-09-13T16:43:54.559111+0000","last_unstale":"2024-09-13T16:43:56.935690+0000","last_undegraded":"2024-09-13T16:43:56.935690+0000","last_fullsized":"2024-09-13T16:43:56.935690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:40.399883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178451+0000","last_change":"2024-09-13T16:43:54.988313+0000","last_active":"2024-09-13T16:43:59.178451+0000","last_peered":"2024-09-13T16:43:59.178451+0000","last_clean":"2024-09-13T16:43:59.178451+0000","last_became_active":"2024-09-13T16:43:54.988059+0000","last_became_peered":"2024-09-13T16:43:54.988059+0000","last_unstale":"2024-09-13T16:43:59.178451+0000","last_undegraded":"2024-09-13T16:43:59.178451+0000","last_fullsized":"2024-09-13T16:43:59.178451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.985469+0000","last_change":"2024-09-13T16:43:54.566410+0000","last_active":"2024-09-13T16:43:59.985469+0000","last_peered":"2024-09-13T16:43:59.985469+0000","last_clean":"2024-09-13T16:43:59.985469+0000","last_became_active":"2024-09-13T16:43:54.565861+0000","last_became_peered":"2024-09-13T16:43:54.565861+0000","last_unstale":"2024-09-13T16:43:59.985469+0000","last_undegraded":"2024-09-13T16:43:59.985469+0000","last_fullsized":"2024-09-13T16:43:59.985469+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178547+0000","last_change":"2024-09-13T16:43:54.989506+0000","last_active":"2024-09-13T16:43:59.178547+0000","last_peered":"2024-09-13T16:43:59.178547+0000","last_clean":"2024-09-13T16:43:59.178547+0000","last_became_active":"2024-09-13T16:43:54.989233+0000","last_became_peered":"2024-09-13T16:43:54.989233+0000","last_unstale":"2024-09-13T16:43:59.178547+0000","last_undegraded":"2024-09-13T16:43:59.178547+0000","last_fullsized":"2024-09-13T16:43:59.178547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:23.913470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975552+0000","last_change":"2024-09-13T16:43:54.559723+0000","last_active":"2024-09-13T16:43:55.975552+0000","last_peered":"2024-09-13T16:43:55.975552+0000","last_clean":"2024-09-13T16:43:55.975552+0000","last_became_active":"2024-09-13T16:43:54.559353+0000","last_became_peered":"2024-09-13T16:43:54.559353+0000","last_unstale":"2024-09-13T16:43:55.975552+0000","last_undegraded":"2024-09-13T16:43:55.975552+0000","last_fullsized":"2024-09-13T16:43:55.975552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486000+0000","last_change":"2024-09-13T16:43:54.992511+0000","last_active":"2024-09-13T16:43:55.486000+0000","last_peered":"2024-09-13T16:43:55.486000+0000","last_clean":"2024-09-13T16:43:55.486000+0000","last_became_active":"2024-09-13T16:43:54.992249+0000","last_became_peered":"2024-09-13T16:43:54.992249+0000","last_unstale":"2024-09-13T16:43:55.486000+0000","last_undegraded":"2024-09-13T16:43:55.486000+0000","last_fullsized":"2024-09-13T16:43:55.486000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.977044+0000","last_change":"2024-09-13T16:43:54.551390+0000","last_active":"2024-09-13T16:43:54.977044+0000","last_peered":"2024-09-13T16:43:54.977044+0000","last_clean":"2024-09-13T16:43:54.977044+0000","last_became_active":"2024-09-13T16:43:54.547074+0000","last_became_peered":"2024-09-13T16:43:54.547074+0000","last_unstale":"2024-09-13T16:43:54.977044+0000","last_undegraded":"2024-09-13T16:43:54.977044+0000","last_fullsized":"2024-09-13T16:43:54.977044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178318+0000","last_change":"2024-09-13T16:43:54.990458+0000","last_active":"2024-09-13T16:43:59.178318+0000","last_peered":"2024-09-13T16:43:59.178318+0000","last_clean":"2024-09-13T16:43:59.178318+0000","last_became_active":"2024-09-13T16:43:54.990196+0000","last_became_peered":"2024-09-13T16:43:54.990196+0000","last_unstale":"2024-09-13T16:43:59.178318+0000","last_undegraded":"2024-09-13T16:43:59.178318+0000","last_fullsized":"2024-09-13T16:43:59.178318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981787+0000","last_change":"2024-09-13T16:43:51.914639+0000","last_active":"2024-09-13T16:43:54.981787+0000","last_peered":"2024-09-13T16:43:54.981787+0000","last_clean":"2024-09-13T16:43:54.981787+0000","last_became_active":"2024-09-13T16:43:51.913797+0000","last_became_peered":"2024-09-13T16:43:51.913797+0000","last_unstale":"2024-09-13T16:43:54.981787+0000","last_undegraded":"2024-09-13T16:43:54.981787+0000","last_fullsized":"2024-09-13T16:43:54.981787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_clean_scrub_stamp":"2024-09-13T16:43:50.498116+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:12.730911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485969+0000","last_change":"2024-09-13T16:43:54.564153+0000","last_active":"2024-09-13T16:43:55.485969+0000","last_peered":"2024-09-13T16:43:55.485969+0000","last_clean":"2024-09-13T16:43:55.485969+0000","last_became_active":"2024-09-13T16:43:54.563949+0000","last_became_peered":"2024-09-13T16:43:54.563949+0000","last_unstale":"2024-09-13T16:43:55.485969+0000","last_undegraded":"2024-09-13T16:43:55.485969+0000","last_fullsized":"2024-09-13T16:43:55.485969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508836+0000","last_change":"2024-09-13T16:43:54.997576+0000","last_active":"2024-09-13T16:43:56.508836+0000","last_peered":"2024-09-13T16:43:56.508836+0000","last_clean":"2024-09-13T16:43:56.508836+0000","last_became_active":"2024-09-13T16:43:54.997315+0000","last_became_peered":"2024-09-13T16:43:54.997315+0000","last_unstale":"2024-09-13T16:43:56.508836+0000","last_undegraded":"2024-09-13T16:43:56.508836+0000","last_fullsized":"2024-09-13T16:43:56.508836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:17.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981701+0000","last_change":"2024-09-13T16:43:54.564804+0000","last_active":"2024-09-13T16:43:54.981701+0000","last_peered":"2024-09-13T16:43:54.981701+0000","last_clean":"2024-09-13T16:43:54.981701+0000","last_became_active":"2024-09-13T16:43:54.564605+0000","last_became_peered":"2024-09-13T16:43:54.564605+0000","last_unstale":"2024-09-13T16:43:54.981701+0000","last_undegraded":"2024-09-13T16:43:54.981701+0000","last_fullsized":"2024-09-13T16:43:54.981701+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.871801+0000","last_change":"2024-09-13T16:43:54.997884+0000","last_active":"2024-09-13T16:43:56.871801+0000","last_peered":"2024-09-13T16:43:56.871801+0000","last_clean":"2024-09-13T16:43:56.871801+0000","last_became_active":"2024-09-13T16:43:54.997510+0000","last_became_peered":"2024-09-13T16:43:54.997510+0000","last_unstale":"2024-09-13T16:43:56.871801+0000","last_undegraded":"2024-09-13T16:43:56.871801+0000","last_fullsized":"2024-09-13T16:43:56.871801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.976301+0000","last_change":"2024-09-13T16:43:54.566294+0000","last_active":"2024-09-13T16:43:54.976301+0000","last_peered":"2024-09-13T16:43:54.976301+0000","last_clean":"2024-09-13T16:43:54.976301+0000","last_became_active":"2024-09-13T16:43:54.566023+0000","last_became_peered":"2024-09-13T16:43:54.566023+0000","last_unstale":"2024-09-13T16:43:54.976301+0000","last_undegraded":"2024-09-13T16:43:54.976301+0000","last_fullsized":"2024-09-13T16:43:54.976301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975316+0000","last_change":"2024-09-13T16:43:54.981116+0000","last_active":"2024-09-13T16:43:55.975316+0000","last_peered":"2024-09-13T16:43:55.975316+0000","last_clean":"2024-09-13T16:43:55.975316+0000","last_became_active":"2024-09-13T16:43:54.980766+0000","last_became_peered":"2024-09-13T16:43:54.980766+0000","last_unstale":"2024-09-13T16:43:55.975316+0000","last_undegraded":"2024-09-13T16:43:55.975316+0000","last_fullsized":"2024-09-13T16:43:55.975316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485959+0000","last_change":"2024-09-13T16:43:54.563716+0000","last_active":"2024-09-13T16:43:55.485959+0000","last_peered":"2024-09-13T16:43:55.485959+0000","last_clean":"2024-09-13T16:43:55.485959+0000","last_became_active":"2024-09-13T16:43:54.563522+0000","last_became_peered":"2024-09-13T16:43:54.563522+0000","last_unstale":"2024-09-13T16:43:55.485959+0000","last_undegraded":"2024-09-13T16:43:55.485959+0000","last_fullsized":"2024-09-13T16:43:55.485959+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935617+0000","last_change":"2024-09-13T16:43:54.999051+0000","last_active":"2024-09-13T16:43:56.935617+0000","last_peered":"2024-09-13T16:43:56.935617+0000","last_clean":"2024-09-13T16:43:56.935617+0000","last_became_active":"2024-09-13T16:43:54.998803+0000","last_became_peered":"2024-09-13T16:43:54.998803+0000","last_unstale":"2024-09-13T16:43:56.935617+0000","last_undegraded":"2024-09-13T16:43:56.935617+0000","last_fullsized":"2024-09-13T16:43:56.935617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.974615+0000","last_change":"2024-09-13T16:43:54.569945+0000","last_active":"2024-09-13T16:43:54.974615+0000","last_peered":"2024-09-13T16:43:54.974615+0000","last_clean":"2024-09-13T16:43:54.974615+0000","last_became_active":"2024-09-13T16:43:54.568846+0000","last_became_peered":"2024-09-13T16:43:54.568846+0000","last_unstale":"2024-09-13T16:43:54.974615+0000","last_undegraded":"2024-09-13T16:43:54.974615+0000","last_fullsized":"2024-09-13T16:43:54.974615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486021+0000","last_change":"2024-09-13T16:43:54.992082+0000","last_active":"2024-09-13T16:43:55.486021+0000","last_peered":"2024-09-13T16:43:55.486021+0000","last_clean":"2024-09-13T16:43:55.486021+0000","last_became_active":"2024-09-13T16:43:54.991659+0000","last_became_peered":"2024-09-13T16:43:54.991659+0000","last_unstale":"2024-09-13T16:43:55.486021+0000","last_undegraded":"2024-09-13T16:43:55.486021+0000","last_fullsized":"2024-09-13T16:43:55.486021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981345+0000","last_change":"2024-09-13T16:43:54.569111+0000","last_active":"2024-09-13T16:43:54.981345+0000","last_peered":"2024-09-13T16:43:54.981345+0000","last_clean":"2024-09-13T16:43:54.981345+0000","last_became_active":"2024-09-13T16:43:54.568903+0000","last_became_peered":"2024-09-13T16:43:54.568903+0000","last_unstale":"2024-09-13T16:43:54.981345+0000","last_undegraded":"2024-09-13T16:43:54.981345+0000","last_fullsized":"2024-09-13T16:43:54.981345+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975475+0000","last_change":"2024-09-13T16:43:54.980100+0000","last_active":"2024-09-13T16:43:55.975475+0000","last_peered":"2024-09-13T16:43:55.975475+0000","last_clean":"2024-09-13T16:43:55.975475+0000","last_became_active":"2024-09-13T16:43:54.979584+0000","last_became_peered":"2024-09-13T16:43:54.979584+0000","last_unstale":"2024-09-13T16:43:55.975475+0000","last_undegraded":"2024-09-13T16:43:55.975475+0000","last_fullsized":"2024-09-13T16:43:55.975475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.975232+0000","last_change":"2024-09-13T16:43:54.560710+0000","last_active":"2024-09-13T16:43:54.975232+0000","last_peered":"2024-09-13T16:43:54.975232+0000","last_clean":"2024-09-13T16:43:54.975232+0000","last_became_active":"2024-09-13T16:43:54.560030+0000","last_became_peered":"2024-09-13T16:43:54.560030+0000","last_unstale":"2024-09-13T16:43:54.975232+0000","last_undegraded":"2024-09-13T16:43:54.975232+0000","last_fullsized":"2024-09-13T16:43:54.975232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.509048+0000","last_change":"2024-09-13T16:43:54.990812+0000","last_active":"2024-09-13T16:43:56.509048+0000","last_peered":"2024-09-13T16:43:56.509048+0000","last_clean":"2024-09-13T16:43:56.509048+0000","last_became_active":"2024-09-13T16:43:54.990440+0000","last_became_peered":"2024-09-13T16:43:54.990440+0000","last_unstale":"2024-09-13T16:43:56.509048+0000","last_undegraded":"2024-09-13T16:43:56.509048+0000","last_fullsized":"2024-09-13T16:43:56.509048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.980132+0000","last_change":"2024-09-13T16:43:54.558439+0000","last_active":"2024-09-13T16:43:59.980132+0000","last_peered":"2024-09-13T16:43:59.980132+0000","last_clean":"2024-09-13T16:43:59.980132+0000","last_became_active":"2024-09-13T16:43:54.558194+0000","last_became_peered":"2024-09-13T16:43:54.558194+0000","last_unstale":"2024-09-13T16:43:59.980132+0000","last_undegraded":"2024-09-13T16:43:59.980132+0000","last_fullsized":"2024-09-13T16:43:59.980132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872087+0000","last_change":"2024-09-13T16:43:54.995534+0000","last_active":"2024-09-13T16:43:56.872087+0000","last_peered":"2024-09-13T16:43:56.872087+0000","last_clean":"2024-09-13T16:43:56.872087+0000","last_became_active":"2024-09-13T16:43:54.995397+0000","last_became_peered":"2024-09-13T16:43:54.995397+0000","last_unstale":"2024-09-13T16:43:56.872087+0000","last_undegraded":"2024-09-13T16:43:56.872087+0000","last_fullsized":"2024-09-13T16:43:56.872087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:19.697247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981614+0000","last_change":"2024-09-13T16:43:54.569085+0000","last_active":"2024-09-13T16:43:54.981614+0000","last_peered":"2024-09-13T16:43:54.981614+0000","last_clean":"2024-09-13T16:43:54.981614+0000","last_became_active":"2024-09-13T16:43:54.568892+0000","last_became_peered":"2024-09-13T16:43:54.568892+0000","last_unstale":"2024-09-13T16:43:54.981614+0000","last_undegraded":"2024-09-13T16:43:54.981614+0000","last_fullsized":"2024-09-13T16:43:54.981614+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935459+0000","last_change":"2024-09-13T16:43:54.984120+0000","last_active":"2024-09-13T16:43:56.935459+0000","last_peered":"2024-09-13T16:43:56.935459+0000","last_clean":"2024-09-13T16:43:56.935459+0000","last_became_active":"2024-09-13T16:43:54.983886+0000","last_became_peered":"2024-09-13T16:43:54.983886+0000","last_unstale":"2024-09-13T16:43:56.935459+0000","last_undegraded":"2024-09-13T16:43:56.935459+0000","last_fullsized":"2024-09-13T16:43:56.935459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.973946+0000","last_change":"2024-09-13T16:43:54.555092+0000","last_active":"2024-09-13T16:43:54.973946+0000","last_peered":"2024-09-13T16:43:54.973946+0000","last_clean":"2024-09-13T16:43:54.973946+0000","last_became_active":"2024-09-13T16:43:54.554230+0000","last_became_peered":"2024-09-13T16:43:54.554230+0000","last_unstale":"2024-09-13T16:43:54.973946+0000","last_undegraded":"2024-09-13T16:43:54.973946+0000","last_fullsized":"2024-09-13T16:43:54.973946+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508681+0000","last_change":"2024-09-13T16:43:54.984588+0000","last_active":"2024-09-13T16:43:56.508681+0000","last_peered":"2024-09-13T16:43:56.508681+0000","last_clean":"2024-09-13T16:43:56.508681+0000","last_became_active":"2024-09-13T16:43:54.984221+0000","last_became_peered":"2024-09-13T16:43:54.984221+0000","last_unstale":"2024-09-13T16:43:56.508681+0000","last_undegraded":"2024-09-13T16:43:56.508681+0000","last_fullsized":"2024-09-13T16:43:56.508681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.973687+0000","last_change":"2024-09-13T16:43:54.562646+0000","last_active":"2024-09-13T16:43:54.973687+0000","last_peered":"2024-09-13T16:43:54.973687+0000","last_clean":"2024-09-13T16:43:54.973687+0000","last_became_active":"2024-09-13T16:43:54.556888+0000","last_became_peered":"2024-09-13T16:43:54.556888+0000","last_unstale":"2024-09-13T16:43:54.973687+0000","last_undegraded":"2024-09-13T16:43:54.973687+0000","last_fullsized":"2024-09-13T16:43:54.973687+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508792+0000","last_change":"2024-09-13T16:43:54.999328+0000","last_active":"2024-09-13T16:43:56.508792+0000","last_peered":"2024-09-13T16:43:56.508792+0000","last_clean":"2024-09-13T16:43:56.508792+0000","last_became_active":"2024-09-13T16:43:54.999101+0000","last_became_peered":"2024-09-13T16:43:54.999101+0000","last_unstale":"2024-09-13T16:43:56.508792+0000","last_undegraded":"2024-09-13T16:43:56.508792+0000","last_fullsized":"2024-09-13T16:43:56.508792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.979268+0000","last_change":"2024-09-13T16:43:54.559214+0000","last_active":"2024-09-13T16:43:59.979268+0000","last_peered":"2024-09-13T16:43:59.979268+0000","last_clean":"2024-09-13T16:43:59.979268+0000","last_became_active":"2024-09-13T16:43:54.558233+0000","last_became_peered":"2024-09-13T16:43:54.558233+0000","last_unstale":"2024-09-13T16:43:59.979268+0000","last_undegraded":"2024-09-13T16:43:59.979268+0000","last_fullsized":"2024-09-13T16:43:59.979268+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260789+0000","last_change":"2024-09-13T16:43:54.986565+0000","last_active":"2024-09-13T16:43:56.260789+0000","last_peered":"2024-09-13T16:43:56.260789+0000","last_clean":"2024-09-13T16:43:56.260789+0000","last_became_active":"2024-09-13T16:43:54.986084+0000","last_became_peered":"2024-09-13T16:43:54.986084+0000","last_unstale":"2024-09-13T16:43:56.260789+0000","last_undegraded":"2024-09-13T16:43:56.260789+0000","last_fullsized":"2024-09-13T16:43:56.260789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486010+0000","last_change":"2024-09-13T16:43:54.982611+0000","last_active":"2024-09-13T16:43:55.486010+0000","last_peered":"2024-09-13T16:43:55.486010+0000","last_clean":"2024-09-13T16:43:55.486010+0000","last_became_active":"2024-09-13T16:43:54.982338+0000","last_became_peered":"2024-09-13T16:43:54.982338+0000","last_unstale":"2024-09-13T16:43:55.486010+0000","last_undegraded":"2024-09-13T16:43:55.486010+0000","last_fullsized":"2024-09-13T16:43:55.486010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.979779+0000","last_change":"2024-09-13T16:43:54.558799+0000","last_active":"2024-09-13T16:43:59.979779+0000","last_peered":"2024-09-13T16:43:59.979779+0000","last_clean":"2024-09-13T16:43:59.979779+0000","last_became_active":"2024-09-13T16:43:54.558332+0000","last_became_peered":"2024-09-13T16:43:54.558332+0000","last_unstale":"2024-09-13T16:43:59.979779+0000","last_undegraded":"2024-09-13T16:43:59.979779+0000","last_fullsized":"2024-09-13T16:43:59.979779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:19.201506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485991+0000","last_change":"2024-09-13T16:43:54.984150+0000","last_active":"2024-09-13T16:43:55.485991+0000","last_peered":"2024-09-13T16:43:55.485991+0000","last_clean":"2024-09-13T16:43:55.485991+0000","last_became_active":"2024-09-13T16:43:54.983881+0000","last_became_peered":"2024-09-13T16:43:54.983881+0000","last_unstale":"2024-09-13T16:43:55.485991+0000","last_undegraded":"2024-09-13T16:43:55.485991+0000","last_fullsized":"2024-09-13T16:43:55.485991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:29.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.980165+0000","last_change":"2024-09-13T16:43:54.559551+0000","last_active":"2024-09-13T16:43:59.980165+0000","last_peered":"2024-09-13T16:43:59.980165+0000","last_clean":"2024-09-13T16:43:59.980165+0000","last_became_active":"2024-09-13T16:43:54.559290+0000","last_became_peered":"2024-09-13T16:43:54.559290+0000","last_unstale":"2024-09-13T16:43:59.980165+0000","last_undegraded":"2024-09-13T16:43:59.980165+0000","last_fullsized":"2024-09-13T16:43:59.980165+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872084+0000","last_change":"2024-09-13T16:43:55.000783+0000","last_active":"2024-09-13T16:43:56.872084+0000","last_peered":"2024-09-13T16:43:56.872084+0000","last_clean":"2024-09-13T16:43:56.872084+0000","last_became_active":"2024-09-13T16:43:55.000196+0000","last_became_peered":"2024-09-13T16:43:55.000196+0000","last_unstale":"2024-09-13T16:43:56.872084+0000","last_undegraded":"2024-09-13T16:43:56.872084+0000","last_fullsized":"2024-09-13T16:43:56.872084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.509030+0000","last_change":"2024-09-13T16:43:54.559886+0000","last_active":"2024-09-13T16:43:56.509030+0000","last_peered":"2024-09-13T16:43:56.509030+0000","last_clean":"2024-09-13T16:43:56.509030+0000","last_became_active":"2024-09-13T16:43:54.559586+0000","last_became_peered":"2024-09-13T16:43:54.559586+0000","last_unstale":"2024-09-13T16:43:56.509030+0000","last_undegraded":"2024-09-13T16:43:56.509030+0000","last_fullsized":"2024-09-13T16:43:56.509030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508978+0000","last_change":"2024-09-13T16:43:54.996099+0000","last_active":"2024-09-13T16:43:56.508978+0000","last_peered":"2024-09-13T16:43:56.508978+0000","last_clean":"2024-09-13T16:43:56.508978+0000","last_became_active":"2024-09-13T16:43:54.995802+0000","last_became_peered":"2024-09-13T16:43:54.995802+0000","last_unstale":"2024-09-13T16:43:56.508978+0000","last_undegraded":"2024-09-13T16:43:56.508978+0000","last_fullsized":"2024-09-13T16:43:56.508978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.986587+0000","last_change":"2024-09-13T16:43:54.562699+0000","last_active":"2024-09-13T16:43:59.986587+0000","last_peered":"2024-09-13T16:43:59.986587+0000","last_clean":"2024-09-13T16:43:59.986587+0000","last_became_active":"2024-09-13T16:43:54.560552+0000","last_became_peered":"2024-09-13T16:43:54.560552+0000","last_unstale":"2024-09-13T16:43:59.986587+0000","last_undegraded":"2024-09-13T16:43:59.986587+0000","last_fullsized":"2024-09-13T16:43:59.986587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:11.032382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":661848,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":70374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1060000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":200704,"data_stored":70284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":278528,"data_stored":660885,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:44:06.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:44:06.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.395+0000 7f28d2e00640 1 Processor -- start 2024-09-13T16:44:06.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.395+0000 7f28d2e00640 1 -- start start 2024-09-13T16:44:06.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.396+0000 7f28d2e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f28cc152da0 0x7f28cc0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:06.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.396+0000 7f28d2e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28cc1593a0 con 0x7f28cc06f0e0 2024-09-13T16:44:06.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.396+0000 7f28d2e00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28cc158ab0 con 0x7f28cc0cced0 2024-09-13T16:44:06.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.396+0000 7f28d2e00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28cc0c1640 con 0x7f28cc152da0 2024-09-13T16:44:06.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.397+0000 7f28d1000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f28cc06f0e0 0x7f28cc0c8240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59596/0 (socket says 172.21.15.28:59596) 2024-09-13T16:44:06.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.397+0000 7f28d1000640 1 -- 172.21.15.28:0/105384229 learned_addr learned my addr 172.21.15.28:0/105384229 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:06.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.397+0000 7f28d1a00640 1 --2- 172.21.15.28:0/105384229 >> v2:172.21.15.191:3300/0 conn(0x7f28cc152da0 0x7f28cc0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:06.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.397+0000 7f28cbe00640 1 -- 172.21.15.28:0/105384229 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574400062 0 0) 0x7f28cc1593a0 con 0x7f28cc06f0e0 2024-09-13T16:44:06.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.397+0000 7f28cbe00640 1 -- 172.21.15.28:0/105384229 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28b4003520 con 0x7f28cc06f0e0 2024-09-13T16:44:06.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.397+0000 7f28d1a00640 1 -- 172.21.15.28:0/105384229 >> v1:172.21.15.191:6789/0 conn(0x7f28cc0cced0 legacy=0x7f28cc163320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:06.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.397+0000 7f28cbe00640 1 -- 172.21.15.28:0/105384229 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843226984 0 0) 0x7f28cc158ab0 con 0x7f28cc0cced0 2024-09-13T16:44:06.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.397+0000 7f28d1a00640 1 -- 172.21.15.28:0/105384229 >> v1:172.21.15.28:6789/0 conn(0x7f28cc06f0e0 legacy=0x7f28cc0c8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:06.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.397+0000 7f28d1a00640 1 -- 172.21.15.28:0/105384229 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28cc0c6f20 con 0x7f28cc152da0 2024-09-13T16:44:06.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.398+0000 7f28d1a00640 1 --2- 172.21.15.28:0/105384229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28cc152da0 0x7f28cc0cc990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f28c00097a0 tx=0x7f28c002a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4a6db5c7a7f7aea server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:06.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.399+0000 7f28cbe00640 1 -- 172.21.15.28:0/105384229 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0032640 con 0x7f28cc152da0 2024-09-13T16:44:06.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.399+0000 7f28cbe00640 1 -- 172.21.15.28:0/105384229 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c00327a0 con 0x7f28cc152da0 2024-09-13T16:44:06.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.399+0000 7f28cbe00640 1 -- 172.21.15.28:0/105384229 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0032b30 con 0x7f28cc152da0 2024-09-13T16:44:06.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.399+0000 7f28d2e00640 1 -- 172.21.15.28:0/105384229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28cc152da0 msgr2=0x7f28cc0cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:06.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.399+0000 7f28d2e00640 1 --2- 172.21.15.28:0/105384229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28cc152da0 0x7f28cc0cc990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f28c00097a0 tx=0x7f28c002a2c0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.399+0000 7f28d2e00640 1 -- 172.21.15.28:0/105384229 shutdown_connections 2024-09-13T16:44:06.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.399+0000 7f28d2e00640 1 --2- 172.21.15.28:0/105384229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28cc152da0 0x7f28cc0cc990 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.399+0000 7f28d2e00640 1 -- 172.21.15.28:0/105384229 >> 172.21.15.28:0/105384229 conn(0x7f28cc066070 msgr2=0x7f28cc066480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:06.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.399+0000 7f28d2e00640 1 -- 172.21.15.28:0/105384229 shutdown_connections 2024-09-13T16:44:06.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.399+0000 7f28d2e00640 1 -- 172.21.15.28:0/105384229 wait complete. 2024-09-13T16:44:06.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.401+0000 7f28d2e00640 1 Processor -- start 2024-09-13T16:44:06.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.401+0000 7f28d2e00640 1 -- start start 2024-09-13T16:44:06.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.402+0000 7f28d2e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f28cc06f0e0 0x7f28cc154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:06.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.402+0000 7f28d2e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28cc0cced0 0x7f28cc0ca570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:06.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.402+0000 7f28d2e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28cc152da0 0x7f28cc1ee870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:06.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.402+0000 7f28d2e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc156570 con 0x7f28cc0cced0 2024-09-13T16:44:06.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.402+0000 7f28d2e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc0c5fb0 con 0x7f28cc152da0 2024-09-13T16:44:06.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.402+0000 7f28d2e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc0c6290 con 0x7f28cc06f0e0 2024-09-13T16:44:06.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.402+0000 7f28d1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28cc0cced0 0x7f28cc0ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:06.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.402+0000 7f28d1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28cc0cced0 0x7f28cc0ca570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41796/0 (socket says 172.21.15.28:41796) 2024-09-13T16:44:06.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.402+0000 7f28d1000640 1 -- 172.21.15.28:0/1089573655 learned_addr learned my addr 172.21.15.28:0/1089573655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:06.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.402+0000 7f28d1a00640 1 --2- 172.21.15.28:0/1089573655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f28cc06f0e0 0x7f28cc154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:06.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.402+0000 7f28d2400640 1 --2- 172.21.15.28:0/1089573655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28cc152da0 0x7f28cc1ee870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:06.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.403+0000 7f28d1000640 1 -- 172.21.15.28:0/1089573655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f28cc06f0e0 msgr2=0x7f28cc154ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:06.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.403+0000 7f28d1000640 1 --2- 172.21.15.28:0/1089573655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f28cc06f0e0 0x7f28cc154ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.403+0000 7f28d1000640 1 -- 172.21.15.28:0/1089573655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28cc152da0 msgr2=0x7f28cc1ee870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.403+0000 7f28d1000640 1 --2- 172.21.15.28:0/1089573655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28cc152da0 0x7f28cc1ee870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.403+0000 7f28d1000640 1 -- 172.21.15.28:0/1089573655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c0009400 con 0x7f28cc0cced0 2024-09-13T16:44:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.403+0000 7f28d1000640 1 --2- 172.21.15.28:0/1089573655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28cc0cced0 0x7f28cc0ca570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f28b800c430 tx=0x7f28b800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.403+0000 7f28c9600640 1 -- 172.21.15.28:0/1089573655 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8005ce0 con 0x7f28cc0cced0 2024-09-13T16:44:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.403+0000 7f28c9600640 1 -- 172.21.15.28:0/1089573655 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b8005e40 con 0x7f28cc0cced0 2024-09-13T16:44:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.403+0000 7f28c9600640 1 -- 172.21.15.28:0/1089573655 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b80072f0 con 0x7f28cc0cced0 2024-09-13T16:44:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.403+0000 7f28d2e00640 1 -- 172.21.15.28:0/1089573655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28cc155c40 con 0x7f28cc0cced0 2024-09-13T16:44:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.403+0000 7f28d2e00640 1 -- 172.21.15.28:0/1089573655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28cc156200 con 0x7f28cc0cced0 2024-09-13T16:44:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.405+0000 7f28d2e00640 1 -- 172.21.15.28:0/1089573655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28cc1ee610 con 0x7f28cc0cced0 2024-09-13T16:44:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.405+0000 7f28c9600640 1 -- 172.21.15.28:0/1089573655 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f28b8007450 con 0x7f28cc0cced0 2024-09-13T16:44:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.406+0000 7f28c9600640 1 --2- 172.21.15.28:0/1089573655 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f28ac06ac00 0x7f28ac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:06.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.406+0000 7f28c9600640 1 -- 172.21.15.28:0/1089573655 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28b80889f0 con 0x7f28cc0cced0 2024-09-13T16:44:06.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.406+0000 7f28d1a00640 1 --2- 172.21.15.28:0/1089573655 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f28ac06ac00 0x7f28ac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:06.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.407+0000 7f28d1a00640 1 --2- 172.21.15.28:0/1089573655 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f28ac06ac00 0x7f28ac06d0c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f28c0034000 tx=0x7f28c002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:06.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.410+0000 7f28c9600640 1 -- 172.21.15.28:0/1089573655 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f28b8053bf0 con 0x7f28cc0cced0 2024-09-13T16:44:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.587+0000 7f28d2e00640 1 -- 172.21.15.28:0/1089573655 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f28cc1584a0 con 0x7f28ac06ac00 2024-09-13T16:44:06.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.594+0000 7f28c9600640 1 -- 172.21.15.28:0/1089573655 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264696 (secure 0 0 0) 0x7f28cc1584a0 con 0x7f28ac06ac00 2024-09-13T16:44:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T16:44:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.598+0000 7f28d2e00640 1 -- 172.21.15.28:0/1089573655 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f28ac06ac00 msgr2=0x7f28ac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.598+0000 7f28d2e00640 1 --2- 172.21.15.28:0/1089573655 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f28ac06ac00 0x7f28ac06d0c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f28c0034000 tx=0x7f28c002e000 comp rx=0 tx=0).stop 2024-09-13T16:44:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.598+0000 7f28d2e00640 1 -- 172.21.15.28:0/1089573655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28cc0cced0 msgr2=0x7f28cc0ca570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.598+0000 7f28d2e00640 1 --2- 172.21.15.28:0/1089573655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28cc0cced0 0x7f28cc0ca570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f28b800c430 tx=0x7f28b800c8f0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.598+0000 7f28d2e00640 1 -- 172.21.15.28:0/1089573655 shutdown_connections 2024-09-13T16:44:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.598+0000 7f28d2e00640 1 --2- 172.21.15.28:0/1089573655 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f28ac06ac00 0x7f28ac06d0c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.598+0000 7f28d2e00640 1 --2- 172.21.15.28:0/1089573655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f28cc152da0 0x7f28cc1ee870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.598+0000 7f28d2e00640 1 --2- 172.21.15.28:0/1089573655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28cc0cced0 0x7f28cc0ca570 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.598+0000 7f28d2e00640 1 --2- 172.21.15.28:0/1089573655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f28cc06f0e0 0x7f28cc154ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.598+0000 7f28d2e00640 1 -- 172.21.15.28:0/1089573655 >> 172.21.15.28:0/1089573655 conn(0x7f28cc066070 msgr2=0x7f28cc157d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.599+0000 7f28d2e00640 1 -- 172.21.15.28:0/1089573655 shutdown_connections 2024-09-13T16:44:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.599+0000 7f28d2e00640 1 -- 172.21.15.28:0/1089573655 wait complete. 2024-09-13T16:44:06.615 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-13T16:44:05.592829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222004,"kb_used_data":1780,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754752716,"statfs":{"total":773094113280,"available":772866781184,"internally_reserved":0,"allocated":1822720,"data_stored":1750397,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12743,"internal_metadata":225431097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002550"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178474+0000","last_change":"2024-09-13T16:43:54.987241+0000","last_active":"2024-09-13T16:43:59.178474+0000","last_peered":"2024-09-13T16:43:59.178474+0000","last_clean":"2024-09-13T16:43:59.178474+0000","last_became_active":"2024-09-13T16:43:54.986988+0000","last_became_peered":"2024-09-13T16:43:54.986988+0000","last_unstale":"2024-09-13T16:43:59.178474+0000","last_undegraded":"2024-09-13T16:43:59.178474+0000","last_fullsized":"2024-09-13T16:43:59.178474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975685+0000","last_change":"2024-09-13T16:43:55.001141+0000","last_active":"2024-09-13T16:43:55.975685+0000","last_peered":"2024-09-13T16:43:55.975685+0000","last_clean":"2024-09-13T16:43:55.975685+0000","last_became_active":"2024-09-13T16:43:55.000757+0000","last_became_peered":"2024-09-13T16:43:55.000757+0000","last_unstale":"2024-09-13T16:43:55.975685+0000","last_undegraded":"2024-09-13T16:43:55.975685+0000","last_fullsized":"2024-09-13T16:43:55.975685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872108+0000","last_change":"2024-09-13T16:43:54.986925+0000","last_active":"2024-09-13T16:43:56.872108+0000","last_peered":"2024-09-13T16:43:56.872108+0000","last_clean":"2024-09-13T16:43:56.872108+0000","last_became_active":"2024-09-13T16:43:54.986198+0000","last_became_peered":"2024-09-13T16:43:54.986198+0000","last_unstale":"2024-09-13T16:43:56.872108+0000","last_undegraded":"2024-09-13T16:43:56.872108+0000","last_fullsized":"2024-09-13T16:43:56.872108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:03.127550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485981+0000","last_change":"2024-09-13T16:43:54.992249+0000","last_active":"2024-09-13T16:43:55.485981+0000","last_peered":"2024-09-13T16:43:55.485981+0000","last_clean":"2024-09-13T16:43:55.485981+0000","last_became_active":"2024-09-13T16:43:54.991804+0000","last_became_peered":"2024-09-13T16:43:54.991804+0000","last_unstale":"2024-09-13T16:43:55.485981+0000","last_undegraded":"2024-09-13T16:43:55.485981+0000","last_fullsized":"2024-09-13T16:43:55.485981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178163+0000","last_change":"2024-09-13T16:43:54.987564+0000","last_active":"2024-09-13T16:43:59.178163+0000","last_peered":"2024-09-13T16:43:59.178163+0000","last_clean":"2024-09-13T16:43:59.178163+0000","last_became_active":"2024-09-13T16:43:54.987328+0000","last_became_peered":"2024-09-13T16:43:54.987328+0000","last_unstale":"2024-09-13T16:43:59.178163+0000","last_undegraded":"2024-09-13T16:43:59.178163+0000","last_fullsized":"2024-09-13T16:43:59.178163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:36.006730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485919+0000","last_change":"2024-09-13T16:43:55.000019+0000","last_active":"2024-09-13T16:43:55.485919+0000","last_peered":"2024-09-13T16:43:55.485919+0000","last_clean":"2024-09-13T16:43:55.485919+0000","last_became_active":"2024-09-13T16:43:54.999752+0000","last_became_peered":"2024-09-13T16:43:54.999752+0000","last_unstale":"2024-09-13T16:43:55.485919+0000","last_undegraded":"2024-09-13T16:43:55.485919+0000","last_fullsized":"2024-09-13T16:43:55.485919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.177985+0000","last_change":"2024-09-13T16:43:54.999533+0000","last_active":"2024-09-13T16:43:59.177985+0000","last_peered":"2024-09-13T16:43:59.177985+0000","last_clean":"2024-09-13T16:43:59.177985+0000","last_became_active":"2024-09-13T16:43:54.999350+0000","last_became_peered":"2024-09-13T16:43:54.999350+0000","last_unstale":"2024-09-13T16:43:59.177985+0000","last_undegraded":"2024-09-13T16:43:59.177985+0000","last_fullsized":"2024-09-13T16:43:59.177985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:29.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508531+0000","last_change":"2024-09-13T16:43:54.998526+0000","last_active":"2024-09-13T16:43:56.508531+0000","last_peered":"2024-09-13T16:43:56.508531+0000","last_clean":"2024-09-13T16:43:56.508531+0000","last_became_active":"2024-09-13T16:43:54.996309+0000","last_became_peered":"2024-09-13T16:43:54.996309+0000","last_unstale":"2024-09-13T16:43:56.508531+0000","last_undegraded":"2024-09-13T16:43:56.508531+0000","last_fullsized":"2024-09-13T16:43:56.508531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178473+0000","last_change":"2024-09-13T16:43:54.993786+0000","last_active":"2024-09-13T16:43:59.178473+0000","last_peered":"2024-09-13T16:43:59.178473+0000","last_clean":"2024-09-13T16:43:59.178473+0000","last_became_active":"2024-09-13T16:43:54.993475+0000","last_became_peered":"2024-09-13T16:43:54.993475+0000","last_unstale":"2024-09-13T16:43:59.178473+0000","last_undegraded":"2024-09-13T16:43:59.178473+0000","last_fullsized":"2024-09-13T16:43:59.178473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260988+0000","last_change":"2024-09-13T16:43:54.986272+0000","last_active":"2024-09-13T16:43:56.260988+0000","last_peered":"2024-09-13T16:43:56.260988+0000","last_clean":"2024-09-13T16:43:56.260988+0000","last_became_active":"2024-09-13T16:43:54.985979+0000","last_became_peered":"2024-09-13T16:43:54.985979+0000","last_unstale":"2024-09-13T16:43:56.260988+0000","last_undegraded":"2024-09-13T16:43:56.260988+0000","last_fullsized":"2024-09-13T16:43:56.260988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260766+0000","last_change":"2024-09-13T16:43:54.986207+0000","last_active":"2024-09-13T16:43:56.260766+0000","last_peered":"2024-09-13T16:43:56.260766+0000","last_clean":"2024-09-13T16:43:56.260766+0000","last_became_active":"2024-09-13T16:43:54.985948+0000","last_became_peered":"2024-09-13T16:43:54.985948+0000","last_unstale":"2024-09-13T16:43:56.260766+0000","last_undegraded":"2024-09-13T16:43:56.260766+0000","last_fullsized":"2024-09-13T16:43:56.260766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935591+0000","last_change":"2024-09-13T16:43:54.986636+0000","last_active":"2024-09-13T16:43:56.935591+0000","last_peered":"2024-09-13T16:43:56.935591+0000","last_clean":"2024-09-13T16:43:56.935591+0000","last_became_active":"2024-09-13T16:43:54.986161+0000","last_became_peered":"2024-09-13T16:43:54.986161+0000","last_unstale":"2024-09-13T16:43:56.935591+0000","last_undegraded":"2024-09-13T16:43:56.935591+0000","last_fullsized":"2024-09-13T16:43:56.935591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:09.340330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508720+0000","last_change":"2024-09-13T16:43:54.997783+0000","last_active":"2024-09-13T16:43:56.508720+0000","last_peered":"2024-09-13T16:43:56.508720+0000","last_clean":"2024-09-13T16:43:56.508720+0000","last_became_active":"2024-09-13T16:43:54.997162+0000","last_became_peered":"2024-09-13T16:43:54.997162+0000","last_unstale":"2024-09-13T16:43:56.508720+0000","last_undegraded":"2024-09-13T16:43:56.508720+0000","last_fullsized":"2024-09-13T16:43:56.508720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872216+0000","last_change":"2024-09-13T16:43:54.995262+0000","last_active":"2024-09-13T16:43:56.872216+0000","last_peered":"2024-09-13T16:43:56.872216+0000","last_clean":"2024-09-13T16:43:56.872216+0000","last_became_active":"2024-09-13T16:43:54.994891+0000","last_became_peered":"2024-09-13T16:43:54.994891+0000","last_unstale":"2024-09-13T16:43:56.872216+0000","last_undegraded":"2024-09-13T16:43:56.872216+0000","last_fullsized":"2024-09-13T16:43:56.872216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975286+0000","last_change":"2024-09-13T16:43:54.981534+0000","last_active":"2024-09-13T16:43:55.975286+0000","last_peered":"2024-09-13T16:43:55.975286+0000","last_clean":"2024-09-13T16:43:55.975286+0000","last_became_active":"2024-09-13T16:43:54.981191+0000","last_became_peered":"2024-09-13T16:43:54.981191+0000","last_unstale":"2024-09-13T16:43:55.975286+0000","last_undegraded":"2024-09-13T16:43:55.975286+0000","last_fullsized":"2024-09-13T16:43:55.975286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485880+0000","last_change":"2024-09-13T16:43:54.994157+0000","last_active":"2024-09-13T16:43:55.485880+0000","last_peered":"2024-09-13T16:43:55.485880+0000","last_clean":"2024-09-13T16:43:55.485880+0000","last_became_active":"2024-09-13T16:43:54.993915+0000","last_became_peered":"2024-09-13T16:43:54.993915+0000","last_unstale":"2024-09-13T16:43:55.485880+0000","last_undegraded":"2024-09-13T16:43:55.485880+0000","last_fullsized":"2024-09-13T16:43:55.485880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486053+0000","last_change":"2024-09-13T16:43:54.987328+0000","last_active":"2024-09-13T16:43:55.486053+0000","last_peered":"2024-09-13T16:43:55.486053+0000","last_clean":"2024-09-13T16:43:55.486053+0000","last_became_active":"2024-09-13T16:43:54.987048+0000","last_became_peered":"2024-09-13T16:43:54.987048+0000","last_unstale":"2024-09-13T16:43:55.486053+0000","last_undegraded":"2024-09-13T16:43:55.486053+0000","last_fullsized":"2024-09-13T16:43:55.486053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.979840+0000","last_change":"2024-09-13T16:43:54.985628+0000","last_active":"2024-09-13T16:43:55.979840+0000","last_peered":"2024-09-13T16:43:55.979840+0000","last_clean":"2024-09-13T16:43:55.979840+0000","last_became_active":"2024-09-13T16:43:54.985378+0000","last_became_peered":"2024-09-13T16:43:54.985378+0000","last_unstale":"2024-09-13T16:43:55.979840+0000","last_undegraded":"2024-09-13T16:43:55.979840+0000","last_fullsized":"2024-09-13T16:43:55.979840+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178469+0000","last_change":"2024-09-13T16:43:54.996712+0000","last_active":"2024-09-13T16:43:59.178469+0000","last_peered":"2024-09-13T16:43:59.178469+0000","last_clean":"2024-09-13T16:43:59.178469+0000","last_became_active":"2024-09-13T16:43:54.996523+0000","last_became_peered":"2024-09-13T16:43:54.996523+0000","last_unstale":"2024-09-13T16:43:59.178469+0000","last_undegraded":"2024-09-13T16:43:59.178469+0000","last_fullsized":"2024-09-13T16:43:59.178469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260867+0000","last_change":"2024-09-13T16:43:54.984879+0000","last_active":"2024-09-13T16:43:56.260867+0000","last_peered":"2024-09-13T16:43:56.260867+0000","last_clean":"2024-09-13T16:43:56.260867+0000","last_became_active":"2024-09-13T16:43:54.984595+0000","last_became_peered":"2024-09-13T16:43:54.984595+0000","last_unstale":"2024-09-13T16:43:56.260867+0000","last_undegraded":"2024-09-13T16:43:56.260867+0000","last_fullsized":"2024-09-13T16:43:56.260867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.871914+0000","last_change":"2024-09-13T16:43:54.996457+0000","last_active":"2024-09-13T16:43:56.871914+0000","last_peered":"2024-09-13T16:43:56.871914+0000","last_clean":"2024-09-13T16:43:56.871914+0000","last_became_active":"2024-09-13T16:43:54.996275+0000","last_became_peered":"2024-09-13T16:43:54.996275+0000","last_unstale":"2024-09-13T16:43:56.871914+0000","last_undegraded":"2024-09-13T16:43:56.871914+0000","last_fullsized":"2024-09-13T16:43:56.871914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485929+0000","last_change":"2024-09-13T16:43:54.985368+0000","last_active":"2024-09-13T16:43:55.485929+0000","last_peered":"2024-09-13T16:43:55.485929+0000","last_clean":"2024-09-13T16:43:55.485929+0000","last_became_active":"2024-09-13T16:43:54.985130+0000","last_became_peered":"2024-09-13T16:43:54.985130+0000","last_unstale":"2024-09-13T16:43:55.485929+0000","last_undegraded":"2024-09-13T16:43:55.485929+0000","last_fullsized":"2024-09-13T16:43:55.485929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508531+0000","last_change":"2024-09-13T16:43:54.991862+0000","last_active":"2024-09-13T16:43:56.508531+0000","last_peered":"2024-09-13T16:43:56.508531+0000","last_clean":"2024-09-13T16:43:56.508531+0000","last_became_active":"2024-09-13T16:43:54.991554+0000","last_became_peered":"2024-09-13T16:43:54.991554+0000","last_unstale":"2024-09-13T16:43:56.508531+0000","last_undegraded":"2024-09-13T16:43:56.508531+0000","last_fullsized":"2024-09-13T16:43:56.508531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485896+0000","last_change":"2024-09-13T16:43:54.992241+0000","last_active":"2024-09-13T16:43:55.485896+0000","last_peered":"2024-09-13T16:43:55.485896+0000","last_clean":"2024-09-13T16:43:55.485896+0000","last_became_active":"2024-09-13T16:43:54.991687+0000","last_became_peered":"2024-09-13T16:43:54.991687+0000","last_unstale":"2024-09-13T16:43:55.485896+0000","last_undegraded":"2024-09-13T16:43:55.485896+0000","last_fullsized":"2024-09-13T16:43:55.485896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.509024+0000","last_change":"2024-09-13T16:43:54.991366+0000","last_active":"2024-09-13T16:43:56.509024+0000","last_peered":"2024-09-13T16:43:56.509024+0000","last_clean":"2024-09-13T16:43:56.509024+0000","last_became_active":"2024-09-13T16:43:54.991127+0000","last_became_peered":"2024-09-13T16:43:54.991127+0000","last_unstale":"2024-09-13T16:43:56.509024+0000","last_undegraded":"2024-09-13T16:43:56.509024+0000","last_fullsized":"2024-09-13T16:43:56.509024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.261006+0000","last_change":"2024-09-13T16:43:54.985615+0000","last_active":"2024-09-13T16:43:56.261006+0000","last_peered":"2024-09-13T16:43:56.261006+0000","last_clean":"2024-09-13T16:43:56.261006+0000","last_became_active":"2024-09-13T16:43:54.985299+0000","last_became_peered":"2024-09-13T16:43:54.985299+0000","last_unstale":"2024-09-13T16:43:56.261006+0000","last_undegraded":"2024-09-13T16:43:56.261006+0000","last_fullsized":"2024-09-13T16:43:56.261006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975382+0000","last_change":"2024-09-13T16:43:54.993325+0000","last_active":"2024-09-13T16:43:55.975382+0000","last_peered":"2024-09-13T16:43:55.975382+0000","last_clean":"2024-09-13T16:43:55.975382+0000","last_became_active":"2024-09-13T16:43:54.993009+0000","last_became_peered":"2024-09-13T16:43:54.993009+0000","last_unstale":"2024-09-13T16:43:55.975382+0000","last_undegraded":"2024-09-13T16:43:55.975382+0000","last_fullsized":"2024-09-13T16:43:55.975382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508820+0000","last_change":"2024-09-13T16:43:54.998229+0000","last_active":"2024-09-13T16:43:56.508820+0000","last_peered":"2024-09-13T16:43:56.508820+0000","last_clean":"2024-09-13T16:43:56.508820+0000","last_became_active":"2024-09-13T16:43:54.995774+0000","last_became_peered":"2024-09-13T16:43:54.995774+0000","last_unstale":"2024-09-13T16:43:56.508820+0000","last_undegraded":"2024-09-13T16:43:56.508820+0000","last_fullsized":"2024-09-13T16:43:56.508820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872235+0000","last_change":"2024-09-13T16:43:55.002126+0000","last_active":"2024-09-13T16:43:56.872235+0000","last_peered":"2024-09-13T16:43:56.872235+0000","last_clean":"2024-09-13T16:43:56.872235+0000","last_became_active":"2024-09-13T16:43:55.001713+0000","last_became_peered":"2024-09-13T16:43:55.001713+0000","last_unstale":"2024-09-13T16:43:56.872235+0000","last_undegraded":"2024-09-13T16:43:56.872235+0000","last_fullsized":"2024-09-13T16:43:56.872235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485939+0000","last_change":"2024-09-13T16:43:54.992745+0000","last_active":"2024-09-13T16:43:55.485939+0000","last_peered":"2024-09-13T16:43:55.485939+0000","last_clean":"2024-09-13T16:43:55.485939+0000","last_became_active":"2024-09-13T16:43:54.992508+0000","last_became_peered":"2024-09-13T16:43:54.992508+0000","last_unstale":"2024-09-13T16:43:55.485939+0000","last_undegraded":"2024-09-13T16:43:55.485939+0000","last_fullsized":"2024-09-13T16:43:55.485939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:23.913470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.979793+0000","last_change":"2024-09-13T16:43:54.990631+0000","last_active":"2024-09-13T16:43:55.979793+0000","last_peered":"2024-09-13T16:43:55.979793+0000","last_clean":"2024-09-13T16:43:55.979793+0000","last_became_active":"2024-09-13T16:43:54.986277+0000","last_became_peered":"2024-09-13T16:43:54.986277+0000","last_unstale":"2024-09-13T16:43:55.979793+0000","last_undegraded":"2024-09-13T16:43:55.979793+0000","last_fullsized":"2024-09-13T16:43:55.979793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260674+0000","last_change":"2024-09-13T16:43:55.000059+0000","last_active":"2024-09-13T16:43:56.260674+0000","last_peered":"2024-09-13T16:43:56.260674+0000","last_clean":"2024-09-13T16:43:56.260674+0000","last_became_active":"2024-09-13T16:43:54.999778+0000","last_became_peered":"2024-09-13T16:43:54.999778+0000","last_unstale":"2024-09-13T16:43:56.260674+0000","last_undegraded":"2024-09-13T16:43:56.260674+0000","last_fullsized":"2024-09-13T16:43:56.260674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935551+0000","last_change":"2024-09-13T16:43:54.988375+0000","last_active":"2024-09-13T16:43:56.935551+0000","last_peered":"2024-09-13T16:43:56.935551+0000","last_clean":"2024-09-13T16:43:56.935551+0000","last_became_active":"2024-09-13T16:43:54.988131+0000","last_became_peered":"2024-09-13T16:43:54.988131+0000","last_unstale":"2024-09-13T16:43:56.935551+0000","last_undegraded":"2024-09-13T16:43:56.935551+0000","last_fullsized":"2024-09-13T16:43:56.935551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.635755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508811+0000","last_change":"2024-09-13T16:43:54.555042+0000","last_active":"2024-09-13T16:43:56.508811+0000","last_peered":"2024-09-13T16:43:56.508811+0000","last_clean":"2024-09-13T16:43:56.508811+0000","last_became_active":"2024-09-13T16:43:54.554697+0000","last_became_peered":"2024-09-13T16:43:54.554697+0000","last_unstale":"2024-09-13T16:43:56.508811+0000","last_undegraded":"2024-09-13T16:43:56.508811+0000","last_fullsized":"2024-09-13T16:43:56.508811+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872075+0000","last_change":"2024-09-13T16:43:54.998755+0000","last_active":"2024-09-13T16:43:56.872075+0000","last_peered":"2024-09-13T16:43:56.872075+0000","last_clean":"2024-09-13T16:43:56.872075+0000","last_became_active":"2024-09-13T16:43:54.998399+0000","last_became_peered":"2024-09-13T16:43:54.998399+0000","last_unstale":"2024-09-13T16:43:56.872075+0000","last_undegraded":"2024-09-13T16:43:56.872075+0000","last_fullsized":"2024-09-13T16:43:56.872075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260801+0000","last_change":"2024-09-13T16:43:54.553250+0000","last_active":"2024-09-13T16:43:56.260801+0000","last_peered":"2024-09-13T16:43:56.260801+0000","last_clean":"2024-09-13T16:43:56.260801+0000","last_became_active":"2024-09-13T16:43:54.552977+0000","last_became_peered":"2024-09-13T16:43:54.552977+0000","last_unstale":"2024-09-13T16:43:56.260801+0000","last_undegraded":"2024-09-13T16:43:56.260801+0000","last_fullsized":"2024-09-13T16:43:56.260801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.984357+0000","last_change":"2024-09-13T16:43:54.558065+0000","last_active":"2024-09-13T16:43:54.984357+0000","last_peered":"2024-09-13T16:43:54.984357+0000","last_clean":"2024-09-13T16:43:54.984357+0000","last_became_active":"2024-09-13T16:43:54.557722+0000","last_became_peered":"2024-09-13T16:43:54.557722+0000","last_unstale":"2024-09-13T16:43:54.984357+0000","last_undegraded":"2024-09-13T16:43:54.984357+0000","last_fullsized":"2024-09-13T16:43:54.984357+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508519+0000","last_change":"2024-09-13T16:43:55.002041+0000","last_active":"2024-09-13T16:43:56.508519+0000","last_peered":"2024-09-13T16:43:56.508519+0000","last_clean":"2024-09-13T16:43:56.508519+0000","last_became_active":"2024-09-13T16:43:55.001512+0000","last_became_peered":"2024-09-13T16:43:55.001512+0000","last_unstale":"2024-09-13T16:43:56.508519+0000","last_undegraded":"2024-09-13T16:43:56.508519+0000","last_fullsized":"2024-09-13T16:43:56.508519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485908+0000","last_change":"2024-09-13T16:43:54.550389+0000","last_active":"2024-09-13T16:43:55.485908+0000","last_peered":"2024-09-13T16:43:55.485908+0000","last_clean":"2024-09-13T16:43:55.485908+0000","last_became_active":"2024-09-13T16:43:54.550154+0000","last_became_peered":"2024-09-13T16:43:54.550154+0000","last_unstale":"2024-09-13T16:43:55.485908+0000","last_undegraded":"2024-09-13T16:43:55.485908+0000","last_fullsized":"2024-09-13T16:43:55.485908+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975641+0000","last_change":"2024-09-13T16:43:54.993377+0000","last_active":"2024-09-13T16:43:55.975641+0000","last_peered":"2024-09-13T16:43:55.975641+0000","last_clean":"2024-09-13T16:43:55.975641+0000","last_became_active":"2024-09-13T16:43:54.993036+0000","last_became_peered":"2024-09-13T16:43:54.993036+0000","last_unstale":"2024-09-13T16:43:55.975641+0000","last_undegraded":"2024-09-13T16:43:55.975641+0000","last_fullsized":"2024-09-13T16:43:55.975641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.978710+0000","last_change":"2024-09-13T16:43:54.555528+0000","last_active":"2024-09-13T16:43:54.978710+0000","last_peered":"2024-09-13T16:43:54.978710+0000","last_clean":"2024-09-13T16:43:54.978710+0000","last_became_active":"2024-09-13T16:43:54.555267+0000","last_became_peered":"2024-09-13T16:43:54.555267+0000","last_unstale":"2024-09-13T16:43:54.978710+0000","last_undegraded":"2024-09-13T16:43:54.978710+0000","last_fullsized":"2024-09-13T16:43:54.978710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486032+0000","last_change":"2024-09-13T16:43:54.981706+0000","last_active":"2024-09-13T16:43:55.486032+0000","last_peered":"2024-09-13T16:43:55.486032+0000","last_clean":"2024-09-13T16:43:55.486032+0000","last_became_active":"2024-09-13T16:43:54.981429+0000","last_became_peered":"2024-09-13T16:43:54.981429+0000","last_unstale":"2024-09-13T16:43:55.486032+0000","last_undegraded":"2024-09-13T16:43:55.486032+0000","last_fullsized":"2024-09-13T16:43:55.486032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486042+0000","last_change":"2024-09-13T16:43:54.554593+0000","last_active":"2024-09-13T16:43:55.486042+0000","last_peered":"2024-09-13T16:43:55.486042+0000","last_clean":"2024-09-13T16:43:55.486042+0000","last_became_active":"2024-09-13T16:43:54.554307+0000","last_became_peered":"2024-09-13T16:43:54.554307+0000","last_unstale":"2024-09-13T16:43:55.486042+0000","last_undegraded":"2024-09-13T16:43:55.486042+0000","last_fullsized":"2024-09-13T16:43:55.486042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872139+0000","last_change":"2024-09-13T16:43:54.993485+0000","last_active":"2024-09-13T16:43:56.872139+0000","last_peered":"2024-09-13T16:43:56.872139+0000","last_clean":"2024-09-13T16:43:56.872139+0000","last_became_active":"2024-09-13T16:43:54.993226+0000","last_became_peered":"2024-09-13T16:43:54.993226+0000","last_unstale":"2024-09-13T16:43:56.872139+0000","last_undegraded":"2024-09-13T16:43:56.872139+0000","last_fullsized":"2024-09-13T16:43:56.872139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.982414+0000","last_change":"2024-09-13T16:43:54.562976+0000","last_active":"2024-09-13T16:43:59.982414+0000","last_peered":"2024-09-13T16:43:59.982414+0000","last_clean":"2024-09-13T16:43:59.982414+0000","last_became_active":"2024-09-13T16:43:54.560480+0000","last_became_peered":"2024-09-13T16:43:54.560480+0000","last_unstale":"2024-09-13T16:43:59.982414+0000","last_undegraded":"2024-09-13T16:43:59.982414+0000","last_fullsized":"2024-09-13T16:43:59.982414+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935689+0000","last_change":"2024-09-13T16:43:54.986459+0000","last_active":"2024-09-13T16:43:56.935689+0000","last_peered":"2024-09-13T16:43:56.935689+0000","last_clean":"2024-09-13T16:43:56.935689+0000","last_became_active":"2024-09-13T16:43:54.986083+0000","last_became_peered":"2024-09-13T16:43:54.986083+0000","last_unstale":"2024-09-13T16:43:56.935689+0000","last_undegraded":"2024-09-13T16:43:56.935689+0000","last_fullsized":"2024-09-13T16:43:56.935689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:51.960459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.985320+0000","last_change":"2024-09-13T16:43:54.565929+0000","last_active":"2024-09-13T16:43:59.985320+0000","last_peered":"2024-09-13T16:43:59.985320+0000","last_clean":"2024-09-13T16:43:59.985320+0000","last_became_active":"2024-09-13T16:43:54.565701+0000","last_became_peered":"2024-09-13T16:43:54.565701+0000","last_unstale":"2024-09-13T16:43:59.985320+0000","last_undegraded":"2024-09-13T16:43:59.985320+0000","last_fullsized":"2024-09-13T16:43:59.985320+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178232+0000","last_change":"2024-09-13T16:43:54.989068+0000","last_active":"2024-09-13T16:43:59.178232+0000","last_peered":"2024-09-13T16:43:59.178232+0000","last_clean":"2024-09-13T16:43:59.178232+0000","last_became_active":"2024-09-13T16:43:54.988779+0000","last_became_peered":"2024-09-13T16:43:54.988779+0000","last_unstale":"2024-09-13T16:43:59.178232+0000","last_undegraded":"2024-09-13T16:43:59.178232+0000","last_fullsized":"2024-09-13T16:43:59.178232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.982285+0000","last_change":"2024-09-13T16:43:54.561487+0000","last_active":"2024-09-13T16:43:59.982285+0000","last_peered":"2024-09-13T16:43:59.982285+0000","last_clean":"2024-09-13T16:43:59.982285+0000","last_became_active":"2024-09-13T16:43:54.561077+0000","last_became_peered":"2024-09-13T16:43:54.561077+0000","last_unstale":"2024-09-13T16:43:59.982285+0000","last_undegraded":"2024-09-13T16:43:59.982285+0000","last_fullsized":"2024-09-13T16:43:59.982285+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872141+0000","last_change":"2024-09-13T16:43:54.999517+0000","last_active":"2024-09-13T16:43:56.872141+0000","last_peered":"2024-09-13T16:43:56.872141+0000","last_clean":"2024-09-13T16:43:56.872141+0000","last_became_active":"2024-09-13T16:43:54.999283+0000","last_became_peered":"2024-09-13T16:43:54.999283+0000","last_unstale":"2024-09-13T16:43:56.872141+0000","last_undegraded":"2024-09-13T16:43:56.872141+0000","last_fullsized":"2024-09-13T16:43:56.872141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:20.207775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485949+0000","last_change":"2024-09-13T16:43:54.542871+0000","last_active":"2024-09-13T16:43:55.485949+0000","last_peered":"2024-09-13T16:43:55.485949+0000","last_clean":"2024-09-13T16:43:55.485949+0000","last_became_active":"2024-09-13T16:43:54.542548+0000","last_became_peered":"2024-09-13T16:43:54.542548+0000","last_unstale":"2024-09-13T16:43:55.485949+0000","last_undegraded":"2024-09-13T16:43:55.485949+0000","last_fullsized":"2024-09-13T16:43:55.485949+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178181+0000","last_change":"2024-09-13T16:43:54.989168+0000","last_active":"2024-09-13T16:43:59.178181+0000","last_peered":"2024-09-13T16:43:59.178181+0000","last_clean":"2024-09-13T16:43:59.178181+0000","last_became_active":"2024-09-13T16:43:54.988716+0000","last_became_peered":"2024-09-13T16:43:54.988716+0000","last_unstale":"2024-09-13T16:43:59.178181+0000","last_undegraded":"2024-09-13T16:43:59.178181+0000","last_fullsized":"2024-09-13T16:43:59.178181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.978600+0000","last_change":"2024-09-13T16:43:54.551551+0000","last_active":"2024-09-13T16:43:54.978600+0000","last_peered":"2024-09-13T16:43:54.978600+0000","last_clean":"2024-09-13T16:43:54.978600+0000","last_became_active":"2024-09-13T16:43:54.551293+0000","last_became_peered":"2024-09-13T16:43:54.551293+0000","last_unstale":"2024-09-13T16:43:54.978600+0000","last_undegraded":"2024-09-13T16:43:54.978600+0000","last_fullsized":"2024-09-13T16:43:54.978600+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260677+0000","last_change":"2024-09-13T16:43:54.994516+0000","last_active":"2024-09-13T16:43:56.260677+0000","last_peered":"2024-09-13T16:43:56.260677+0000","last_clean":"2024-09-13T16:43:56.260677+0000","last_became_active":"2024-09-13T16:43:54.994245+0000","last_became_peered":"2024-09-13T16:43:54.994245+0000","last_unstale":"2024-09-13T16:43:56.260677+0000","last_undegraded":"2024-09-13T16:43:56.260677+0000","last_fullsized":"2024-09-13T16:43:56.260677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.985709+0000","last_change":"2024-09-13T16:43:54.562232+0000","last_active":"2024-09-13T16:43:59.985709+0000","last_peered":"2024-09-13T16:43:59.985709+0000","last_clean":"2024-09-13T16:43:59.985709+0000","last_became_active":"2024-09-13T16:43:54.561939+0000","last_became_peered":"2024-09-13T16:43:54.561939+0000","last_unstale":"2024-09-13T16:43:59.985709+0000","last_undegraded":"2024-09-13T16:43:59.985709+0000","last_fullsized":"2024-09-13T16:43:59.985709+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178047+0000","last_change":"2024-09-13T16:43:54.988439+0000","last_active":"2024-09-13T16:43:59.178047+0000","last_peered":"2024-09-13T16:43:59.178047+0000","last_clean":"2024-09-13T16:43:59.178047+0000","last_became_active":"2024-09-13T16:43:54.988172+0000","last_became_peered":"2024-09-13T16:43:54.988172+0000","last_unstale":"2024-09-13T16:43:59.178047+0000","last_undegraded":"2024-09-13T16:43:59.178047+0000","last_fullsized":"2024-09-13T16:43:59.178047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935690+0000","last_change":"2024-09-13T16:43:54.559372+0000","last_active":"2024-09-13T16:43:56.935690+0000","last_peered":"2024-09-13T16:43:56.935690+0000","last_clean":"2024-09-13T16:43:56.935690+0000","last_became_active":"2024-09-13T16:43:54.559111+0000","last_became_peered":"2024-09-13T16:43:54.559111+0000","last_unstale":"2024-09-13T16:43:56.935690+0000","last_undegraded":"2024-09-13T16:43:56.935690+0000","last_fullsized":"2024-09-13T16:43:56.935690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:40.399883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178451+0000","last_change":"2024-09-13T16:43:54.988313+0000","last_active":"2024-09-13T16:43:59.178451+0000","last_peered":"2024-09-13T16:43:59.178451+0000","last_clean":"2024-09-13T16:43:59.178451+0000","last_became_active":"2024-09-13T16:43:54.988059+0000","last_became_peered":"2024-09-13T16:43:54.988059+0000","last_unstale":"2024-09-13T16:43:59.178451+0000","last_undegraded":"2024-09-13T16:43:59.178451+0000","last_fullsized":"2024-09-13T16:43:59.178451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.985469+0000","last_change":"2024-09-13T16:43:54.566410+0000","last_active":"2024-09-13T16:43:59.985469+0000","last_peered":"2024-09-13T16:43:59.985469+0000","last_clean":"2024-09-13T16:43:59.985469+0000","last_became_active":"2024-09-13T16:43:54.565861+0000","last_became_peered":"2024-09-13T16:43:54.565861+0000","last_unstale":"2024-09-13T16:43:59.985469+0000","last_undegraded":"2024-09-13T16:43:59.985469+0000","last_fullsized":"2024-09-13T16:43:59.985469+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178547+0000","last_change":"2024-09-13T16:43:54.989506+0000","last_active":"2024-09-13T16:43:59.178547+0000","last_peered":"2024-09-13T16:43:59.178547+0000","last_clean":"2024-09-13T16:43:59.178547+0000","last_became_active":"2024-09-13T16:43:54.989233+0000","last_became_peered":"2024-09-13T16:43:54.989233+0000","last_unstale":"2024-09-13T16:43:59.178547+0000","last_undegraded":"2024-09-13T16:43:59.178547+0000","last_fullsized":"2024-09-13T16:43:59.178547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:23.913470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975552+0000","last_change":"2024-09-13T16:43:54.559723+0000","last_active":"2024-09-13T16:43:55.975552+0000","last_peered":"2024-09-13T16:43:55.975552+0000","last_clean":"2024-09-13T16:43:55.975552+0000","last_became_active":"2024-09-13T16:43:54.559353+0000","last_became_peered":"2024-09-13T16:43:54.559353+0000","last_unstale":"2024-09-13T16:43:55.975552+0000","last_undegraded":"2024-09-13T16:43:55.975552+0000","last_fullsized":"2024-09-13T16:43:55.975552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486000+0000","last_change":"2024-09-13T16:43:54.992511+0000","last_active":"2024-09-13T16:43:55.486000+0000","last_peered":"2024-09-13T16:43:55.486000+0000","last_clean":"2024-09-13T16:43:55.486000+0000","last_became_active":"2024-09-13T16:43:54.992249+0000","last_became_peered":"2024-09-13T16:43:54.992249+0000","last_unstale":"2024-09-13T16:43:55.486000+0000","last_undegraded":"2024-09-13T16:43:55.486000+0000","last_fullsized":"2024-09-13T16:43:55.486000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.977044+0000","last_change":"2024-09-13T16:43:54.551390+0000","last_active":"2024-09-13T16:43:54.977044+0000","last_peered":"2024-09-13T16:43:54.977044+0000","last_clean":"2024-09-13T16:43:54.977044+0000","last_became_active":"2024-09-13T16:43:54.547074+0000","last_became_peered":"2024-09-13T16:43:54.547074+0000","last_unstale":"2024-09-13T16:43:54.977044+0000","last_undegraded":"2024-09-13T16:43:54.977044+0000","last_fullsized":"2024-09-13T16:43:54.977044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178318+0000","last_change":"2024-09-13T16:43:54.990458+0000","last_active":"2024-09-13T16:43:59.178318+0000","last_peered":"2024-09-13T16:43:59.178318+0000","last_clean":"2024-09-13T16:43:59.178318+0000","last_became_active":"2024-09-13T16:43:54.990196+0000","last_became_peered":"2024-09-13T16:43:54.990196+0000","last_unstale":"2024-09-13T16:43:59.178318+0000","last_undegraded":"2024-09-13T16:43:59.178318+0000","last_fullsized":"2024-09-13T16:43:59.178318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981787+0000","last_change":"2024-09-13T16:43:51.914639+0000","last_active":"2024-09-13T16:43:54.981787+0000","last_peered":"2024-09-13T16:43:54.981787+0000","last_clean":"2024-09-13T16:43:54.981787+0000","last_became_active":"2024-09-13T16:43:51.913797+0000","last_became_peered":"2024-09-13T16:43:51.913797+0000","last_unstale":"2024-09-13T16:43:54.981787+0000","last_undegraded":"2024-09-13T16:43:54.981787+0000","last_fullsized":"2024-09-13T16:43:54.981787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_clean_scrub_stamp":"2024-09-13T16:43:50.498116+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:12.730911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485969+0000","last_change":"2024-09-13T16:43:54.564153+0000","last_active":"2024-09-13T16:43:55.485969+0000","last_peered":"2024-09-13T16:43:55.485969+0000","last_clean":"2024-09-13T16:43:55.485969+0000","last_became_active":"2024-09-13T16:43:54.563949+0000","last_became_peered":"2024-09-13T16:43:54.563949+0000","last_unstale":"2024-09-13T16:43:55.485969+0000","last_undegraded":"2024-09-13T16:43:55.485969+0000","last_fullsized":"2024-09-13T16:43:55.485969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508836+0000","last_change":"2024-09-13T16:43:54.997576+0000","last_active":"2024-09-13T16:43:56.508836+0000","last_peered":"2024-09-13T16:43:56.508836+0000","last_clean":"2024-09-13T16:43:56.508836+0000","last_became_active":"2024-09-13T16:43:54.997315+0000","last_became_peered":"2024-09-13T16:43:54.997315+0000","last_unstale":"2024-09-13T16:43:56.508836+0000","last_undegraded":"2024-09-13T16:43:56.508836+0000","last_fullsized":"2024-09-13T16:43:56.508836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:17.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981701+0000","last_change":"2024-09-13T16:43:54.564804+0000","last_active":"2024-09-13T16:43:54.981701+0000","last_peered":"2024-09-13T16:43:54.981701+0000","last_clean":"2024-09-13T16:43:54.981701+0000","last_became_active":"2024-09-13T16:43:54.564605+0000","last_became_peered":"2024-09-13T16:43:54.564605+0000","last_unstale":"2024-09-13T16:43:54.981701+0000","last_undegraded":"2024-09-13T16:43:54.981701+0000","last_fullsized":"2024-09-13T16:43:54.981701+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.871801+0000","last_change":"2024-09-13T16:43:54.997884+0000","last_active":"2024-09-13T16:43:56.871801+0000","last_peered":"2024-09-13T16:43:56.871801+0000","last_clean":"2024-09-13T16:43:56.871801+0000","last_became_active":"2024-09-13T16:43:54.997510+0000","last_became_peered":"2024-09-13T16:43:54.997510+0000","last_unstale":"2024-09-13T16:43:56.871801+0000","last_undegraded":"2024-09-13T16:43:56.871801+0000","last_fullsized":"2024-09-13T16:43:56.871801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.976301+0000","last_change":"2024-09-13T16:43:54.566294+0000","last_active":"2024-09-13T16:43:54.976301+0000","last_peered":"2024-09-13T16:43:54.976301+0000","last_clean":"2024-09-13T16:43:54.976301+0000","last_became_active":"2024-09-13T16:43:54.566023+0000","last_became_peered":"2024-09-13T16:43:54.566023+0000","last_unstale":"2024-09-13T16:43:54.976301+0000","last_undegraded":"2024-09-13T16:43:54.976301+0000","last_fullsized":"2024-09-13T16:43:54.976301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975316+0000","last_change":"2024-09-13T16:43:54.981116+0000","last_active":"2024-09-13T16:43:55.975316+0000","last_peered":"2024-09-13T16:43:55.975316+0000","last_clean":"2024-09-13T16:43:55.975316+0000","last_became_active":"2024-09-13T16:43:54.980766+0000","last_became_peered":"2024-09-13T16:43:54.980766+0000","last_unstale":"2024-09-13T16:43:55.975316+0000","last_undegraded":"2024-09-13T16:43:55.975316+0000","last_fullsized":"2024-09-13T16:43:55.975316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485959+0000","last_change":"2024-09-13T16:43:54.563716+0000","last_active":"2024-09-13T16:43:55.485959+0000","last_peered":"2024-09-13T16:43:55.485959+0000","last_clean":"2024-09-13T16:43:55.485959+0000","last_became_active":"2024-09-13T16:43:54.563522+0000","last_became_peered":"2024-09-13T16:43:54.563522+0000","last_unstale":"2024-09-13T16:43:55.485959+0000","last_undegraded":"2024-09-13T16:43:55.485959+0000","last_fullsized":"2024-09-13T16:43:55.485959+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935617+0000","last_change":"2024-09-13T16:43:54.999051+0000","last_active":"2024-09-13T16:43:56.935617+0000","last_peered":"2024-09-13T16:43:56.935617+0000","last_clean":"2024-09-13T16:43:56.935617+0000","last_became_active":"2024-09-13T16:43:54.998803+0000","last_became_peered":"2024-09-13T16:43:54.998803+0000","last_unstale":"2024-09-13T16:43:56.935617+0000","last_undegraded":"2024-09-13T16:43:56.935617+0000","last_fullsized":"2024-09-13T16:43:56.935617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.974615+0000","last_change":"2024-09-13T16:43:54.569945+0000","last_active":"2024-09-13T16:43:54.974615+0000","last_peered":"2024-09-13T16:43:54.974615+0000","last_clean":"2024-09-13T16:43:54.974615+0000","last_became_active":"2024-09-13T16:43:54.568846+0000","last_became_peered":"2024-09-13T16:43:54.568846+0000","last_unstale":"2024-09-13T16:43:54.974615+0000","last_undegraded":"2024-09-13T16:43:54.974615+0000","last_fullsized":"2024-09-13T16:43:54.974615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486021+0000","last_change":"2024-09-13T16:43:54.992082+0000","last_active":"2024-09-13T16:43:55.486021+0000","last_peered":"2024-09-13T16:43:55.486021+0000","last_clean":"2024-09-13T16:43:55.486021+0000","last_became_active":"2024-09-13T16:43:54.991659+0000","last_became_peered":"2024-09-13T16:43:54.991659+0000","last_unstale":"2024-09-13T16:43:55.486021+0000","last_undegraded":"2024-09-13T16:43:55.486021+0000","last_fullsized":"2024-09-13T16:43:55.486021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981345+0000","last_change":"2024-09-13T16:43:54.569111+0000","last_active":"2024-09-13T16:43:54.981345+0000","last_peered":"2024-09-13T16:43:54.981345+0000","last_clean":"2024-09-13T16:43:54.981345+0000","last_became_active":"2024-09-13T16:43:54.568903+0000","last_became_peered":"2024-09-13T16:43:54.568903+0000","last_unstale":"2024-09-13T16:43:54.981345+0000","last_undegraded":"2024-09-13T16:43:54.981345+0000","last_fullsized":"2024-09-13T16:43:54.981345+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975475+0000","last_change":"2024-09-13T16:43:54.980100+0000","last_active":"2024-09-13T16:43:55.975475+0000","last_peered":"2024-09-13T16:43:55.975475+0000","last_clean":"2024-09-13T16:43:55.975475+0000","last_became_active":"2024-09-13T16:43:54.979584+0000","last_became_peered":"2024-09-13T16:43:54.979584+0000","last_unstale":"2024-09-13T16:43:55.975475+0000","last_undegraded":"2024-09-13T16:43:55.975475+0000","last_fullsized":"2024-09-13T16:43:55.975475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.975232+0000","last_change":"2024-09-13T16:43:54.560710+0000","last_active":"2024-09-13T16:43:54.975232+0000","last_peered":"2024-09-13T16:43:54.975232+0000","last_clean":"2024-09-13T16:43:54.975232+0000","last_became_active":"2024-09-13T16:43:54.560030+0000","last_became_peered":"2024-09-13T16:43:54.560030+0000","last_unstale":"2024-09-13T16:43:54.975232+0000","last_undegraded":"2024-09-13T16:43:54.975232+0000","last_fullsized":"2024-09-13T16:43:54.975232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.509048+0000","last_change":"2024-09-13T16:43:54.990812+0000","last_active":"2024-09-13T16:43:56.509048+0000","last_peered":"2024-09-13T16:43:56.509048+0000","last_clean":"2024-09-13T16:43:56.509048+0000","last_became_active":"2024-09-13T16:43:54.990440+0000","last_became_peered":"2024-09-13T16:43:54.990440+0000","last_unstale":"2024-09-13T16:43:56.509048+0000","last_undegraded":"2024-09-13T16:43:56.509048+0000","last_fullsized":"2024-09-13T16:43:56.509048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.980132+0000","last_change":"2024-09-13T16:43:54.558439+0000","last_active":"2024-09-13T16:43:59.980132+0000","last_peered":"2024-09-13T16:43:59.980132+0000","last_clean":"2024-09-13T16:43:59.980132+0000","last_became_active":"2024-09-13T16:43:54.558194+0000","last_became_peered":"2024-09-13T16:43:54.558194+0000","last_unstale":"2024-09-13T16:43:59.980132+0000","last_undegraded":"2024-09-13T16:43:59.980132+0000","last_fullsized":"2024-09-13T16:43:59.980132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872087+0000","last_change":"2024-09-13T16:43:54.995534+0000","last_active":"2024-09-13T16:43:56.872087+0000","last_peered":"2024-09-13T16:43:56.872087+0000","last_clean":"2024-09-13T16:43:56.872087+0000","last_became_active":"2024-09-13T16:43:54.995397+0000","last_became_peered":"2024-09-13T16:43:54.995397+0000","last_unstale":"2024-09-13T16:43:56.872087+0000","last_undegraded":"2024-09-13T16:43:56.872087+0000","last_fullsized":"2024-09-13T16:43:56.872087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:19.697247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981614+0000","last_change":"2024-09-13T16:43:54.569085+0000","last_active":"2024-09-13T16:43:54.981614+0000","last_peered":"2024-09-13T16:43:54.981614+0000","last_clean":"2024-09-13T16:43:54.981614+0000","last_became_active":"2024-09-13T16:43:54.568892+0000","last_became_peered":"2024-09-13T16:43:54.568892+0000","last_unstale":"2024-09-13T16:43:54.981614+0000","last_undegraded":"2024-09-13T16:43:54.981614+0000","last_fullsized":"2024-09-13T16:43:54.981614+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935459+0000","last_change":"2024-09-13T16:43:54.984120+0000","last_active":"2024-09-13T16:43:56.935459+0000","last_peered":"2024-09-13T16:43:56.935459+0000","last_clean":"2024-09-13T16:43:56.935459+0000","last_became_active":"2024-09-13T16:43:54.983886+0000","last_became_peered":"2024-09-13T16:43:54.983886+0000","last_unstale":"2024-09-13T16:43:56.935459+0000","last_undegraded":"2024-09-13T16:43:56.935459+0000","last_fullsized":"2024-09-13T16:43:56.935459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.973946+0000","last_change":"2024-09-13T16:43:54.555092+0000","last_active":"2024-09-13T16:43:54.973946+0000","last_peered":"2024-09-13T16:43:54.973946+0000","last_clean":"2024-09-13T16:43:54.973946+0000","last_became_active":"2024-09-13T16:43:54.554230+0000","last_became_peered":"2024-09-13T16:43:54.554230+0000","last_unstale":"2024-09-13T16:43:54.973946+0000","last_undegraded":"2024-09-13T16:43:54.973946+0000","last_fullsized":"2024-09-13T16:43:54.973946+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508681+0000","last_change":"2024-09-13T16:43:54.984588+0000","last_active":"2024-09-13T16:43:56.508681+0000","last_peered":"2024-09-13T16:43:56.508681+0000","last_clean":"2024-09-13T16:43:56.508681+0000","last_became_active":"2024-09-13T16:43:54.984221+0000","last_became_peered":"2024-09-13T16:43:54.984221+0000","last_unstale":"2024-09-13T16:43:56.508681+0000","last_undegraded":"2024-09-13T16:43:56.508681+0000","last_fullsized":"2024-09-13T16:43:56.508681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.973687+0000","last_change":"2024-09-13T16:43:54.562646+0000","last_active":"2024-09-13T16:43:54.973687+0000","last_peered":"2024-09-13T16:43:54.973687+0000","last_clean":"2024-09-13T16:43:54.973687+0000","last_became_active":"2024-09-13T16:43:54.556888+0000","last_became_peered":"2024-09-13T16:43:54.556888+0000","last_unstale":"2024-09-13T16:43:54.973687+0000","last_undegraded":"2024-09-13T16:43:54.973687+0000","last_fullsized":"2024-09-13T16:43:54.973687+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508792+0000","last_change":"2024-09-13T16:43:54.999328+0000","last_active":"2024-09-13T16:43:56.508792+0000","last_peered":"2024-09-13T16:43:56.508792+0000","last_clean":"2024-09-13T16:43:56.508792+0000","last_became_active":"2024-09-13T16:43:54.999101+0000","last_became_peered":"2024-09-13T16:43:54.999101+0000","last_unstale":"2024-09-13T16:43:56.508792+0000","last_undegraded":"2024-09-13T16:43:56.508792+0000","last_fullsized":"2024-09-13T16:43:56.508792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.979268+0000","last_change":"2024-09-13T16:43:54.559214+0000","last_active":"2024-09-13T16:43:59.979268+0000","last_peered":"2024-09-13T16:43:59.979268+0000","last_clean":"2024-09-13T16:43:59.979268+0000","last_became_active":"2024-09-13T16:43:54.558233+0000","last_became_peered":"2024-09-13T16:43:54.558233+0000","last_unstale":"2024-09-13T16:43:59.979268+0000","last_undegraded":"2024-09-13T16:43:59.979268+0000","last_fullsized":"2024-09-13T16:43:59.979268+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260789+0000","last_change":"2024-09-13T16:43:54.986565+0000","last_active":"2024-09-13T16:43:56.260789+0000","last_peered":"2024-09-13T16:43:56.260789+0000","last_clean":"2024-09-13T16:43:56.260789+0000","last_became_active":"2024-09-13T16:43:54.986084+0000","last_became_peered":"2024-09-13T16:43:54.986084+0000","last_unstale":"2024-09-13T16:43:56.260789+0000","last_undegraded":"2024-09-13T16:43:56.260789+0000","last_fullsized":"2024-09-13T16:43:56.260789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486010+0000","last_change":"2024-09-13T16:43:54.982611+0000","last_active":"2024-09-13T16:43:55.486010+0000","last_peered":"2024-09-13T16:43:55.486010+0000","last_clean":"2024-09-13T16:43:55.486010+0000","last_became_active":"2024-09-13T16:43:54.982338+0000","last_became_peered":"2024-09-13T16:43:54.982338+0000","last_unstale":"2024-09-13T16:43:55.486010+0000","last_undegraded":"2024-09-13T16:43:55.486010+0000","last_fullsized":"2024-09-13T16:43:55.486010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.979779+0000","last_change":"2024-09-13T16:43:54.558799+0000","last_active":"2024-09-13T16:43:59.979779+0000","last_peered":"2024-09-13T16:43:59.979779+0000","last_clean":"2024-09-13T16:43:59.979779+0000","last_became_active":"2024-09-13T16:43:54.558332+0000","last_became_peered":"2024-09-13T16:43:54.558332+0000","last_unstale":"2024-09-13T16:43:59.979779+0000","last_undegraded":"2024-09-13T16:43:59.979779+0000","last_fullsized":"2024-09-13T16:43:59.979779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:19.201506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485991+0000","last_change":"2024-09-13T16:43:54.984150+0000","last_active":"2024-09-13T16:43:55.485991+0000","last_peered":"2024-09-13T16:43:55.485991+0000","last_clean":"2024-09-13T16:43:55.485991+0000","last_became_active":"2024-09-13T16:43:54.983881+0000","last_became_peered":"2024-09-13T16:43:54.983881+0000","last_unstale":"2024-09-13T16:43:55.485991+0000","last_undegraded":"2024-09-13T16:43:55.485991+0000","last_fullsized":"2024-09-13T16:43:55.485991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:29.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.980165+0000","last_change":"2024-09-13T16:43:54.559551+0000","last_active":"2024-09-13T16:43:59.980165+0000","last_peered":"2024-09-13T16:43:59.980165+0000","last_clean":"2024-09-13T16:43:59.980165+0000","last_became_active":"2024-09-13T16:43:54.559290+0000","last_became_peered":"2024-09-13T16:43:54.559290+0000","last_unstale":"2024-09-13T16:43:59.980165+0000","last_undegraded":"2024-09-13T16:43:59.980165+0000","last_fullsized":"2024-09-13T16:43:59.980165+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872084+0000","last_change":"2024-09-13T16:43:55.000783+0000","last_active":"2024-09-13T16:43:56.872084+0000","last_peered":"2024-09-13T16:43:56.872084+0000","last_clean":"2024-09-13T16:43:56.872084+0000","last_became_active":"2024-09-13T16:43:55.000196+0000","last_became_peered":"2024-09-13T16:43:55.000196+0000","last_unstale":"2024-09-13T16:43:56.872084+0000","last_undegraded":"2024-09-13T16:43:56.872084+0000","last_fullsized":"2024-09-13T16:43:56.872084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.509030+0000","last_change":"2024-09-13T16:43:54.559886+0000","last_active":"2024-09-13T16:43:56.509030+0000","last_peered":"2024-09-13T16:43:56.509030+0000","last_clean":"2024-09-13T16:43:56.509030+0000","last_became_active":"2024-09-13T16:43:54.559586+0000","last_became_peered":"2024-09-13T16:43:54.559586+0000","last_unstale":"2024-09-13T16:43:56.509030+0000","last_undegraded":"2024-09-13T16:43:56.509030+0000","last_fullsized":"2024-09-13T16:43:56.509030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508978+0000","last_change":"2024-09-13T16:43:54.996099+0000","last_active":"2024-09-13T16:43:56.508978+0000","last_peered":"2024-09-13T16:43:56.508978+0000","last_clean":"2024-09-13T16:43:56.508978+0000","last_became_active":"2024-09-13T16:43:54.995802+0000","last_became_peered":"2024-09-13T16:43:54.995802+0000","last_unstale":"2024-09-13T16:43:56.508978+0000","last_undegraded":"2024-09-13T16:43:56.508978+0000","last_fullsized":"2024-09-13T16:43:56.508978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.986587+0000","last_change":"2024-09-13T16:43:54.562699+0000","last_active":"2024-09-13T16:43:59.986587+0000","last_peered":"2024-09-13T16:43:59.986587+0000","last_clean":"2024-09-13T16:43:59.986587+0000","last_became_active":"2024-09-13T16:43:54.560552+0000","last_became_peered":"2024-09-13T16:43:54.560552+0000","last_unstale":"2024-09-13T16:43:59.986587+0000","last_undegraded":"2024-09-13T16:43:59.986587+0000","last_fullsized":"2024-09-13T16:43:59.986587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:11.032382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":661848,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":70374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1060000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":200704,"data_stored":70284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":278528,"data_stored":660885,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:44:06.619 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T16:44:06.619 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T16:44:06.619 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-13T16:44:06.620 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T16:44:06.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.807+0000 7f1fb0800640 1 Processor -- start 2024-09-13T16:44:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.807+0000 7f1fb0800640 1 -- start start 2024-09-13T16:44:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.808+0000 7f1fb0800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1fa8152da0 0x7f1fa815c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.808+0000 7f1fb0800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1fa80ca120 0x7f1fa80cb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.808+0000 7f1fb0800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1fa80ca520 0x7f1fa80cb950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.808+0000 7f1fb0800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fa806b130 con 0x7f1fa80ca120 2024-09-13T16:44:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.808+0000 7f1fb0800640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fa80cad30 con 0x7f1fa80ca520 2024-09-13T16:44:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.808+0000 7f1fb0800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa815d060 con 0x7f1fa8152da0 2024-09-13T16:44:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.808+0000 7f1fad400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1fa80ca120 0x7f1fa80cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.808+0000 7f1fae800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1fa80ca520 0x7f1fa80cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.808+0000 7f1fade00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1fa8152da0 0x7f1fa815c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.808+0000 7f1fade00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1fa8152da0 0x7f1fa815c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37754/0 (socket says 172.21.15.28:37754) 2024-09-13T16:44:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.808+0000 7f1fade00640 1 -- 172.21.15.28:0/2836673328 learned_addr learned my addr 172.21.15.28:0/2836673328 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.808+0000 7f1fae800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1fa80ca520 0x7f1fa80cb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60996/0 (socket says 172.21.15.28:60996) 2024-09-13T16:44:06.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.809+0000 7f1fade00640 1 -- 172.21.15.28:0/2836673328 >> v2:172.21.15.191:3300/0 conn(0x7f1fa80ca520 msgr2=0x7f1fa80cb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:06.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.809+0000 7f1fade00640 1 --2- 172.21.15.28:0/2836673328 >> v2:172.21.15.191:3300/0 conn(0x7f1fa80ca520 0x7f1fa80cb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.809+0000 7f1fade00640 1 -- 172.21.15.28:0/2836673328 >> v2:172.21.15.28:3300/0 conn(0x7f1fa80ca120 msgr2=0x7f1fa80cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:06.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.809+0000 7f1fade00640 1 --2- 172.21.15.28:0/2836673328 >> v2:172.21.15.28:3300/0 conn(0x7f1fa80ca120 0x7f1fa80cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.809+0000 7f1fade00640 1 -- 172.21.15.28:0/2836673328 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa815c7c0 con 0x7f1fa8152da0 2024-09-13T16:44:06.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.810+0000 7f1fade00640 1 --2- 172.21.15.28:0/2836673328 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1fa8152da0 0x7f1fa815c130 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80b1ff0 tx=0x7f1f9c02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=608bf6210769b3d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:06.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.810+0000 7f1faca00640 1 -- 172.21.15.28:0/2836673328 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c032440 con 0x7f1fa8152da0 2024-09-13T16:44:06.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.810+0000 7f1faca00640 1 -- 172.21.15.28:0/2836673328 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9c0325a0 con 0x7f1fa8152da0 2024-09-13T16:44:06.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.810+0000 7f1faca00640 1 -- 172.21.15.28:0/2836673328 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c032930 con 0x7f1fa8152da0 2024-09-13T16:44:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.811+0000 7f1fb0800640 1 -- 172.21.15.28:0/2836673328 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1fa8152da0 msgr2=0x7f1fa815c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.811+0000 7f1fb0800640 1 --2- 172.21.15.28:0/2836673328 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1fa8152da0 0x7f1fa815c130 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80b1ff0 tx=0x7f1f9c02a270 comp rx=0 tx=0).stop 2024-09-13T16:44:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.811+0000 7f1fb0800640 1 -- 172.21.15.28:0/2836673328 shutdown_connections 2024-09-13T16:44:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.811+0000 7f1fb0800640 1 --2- 172.21.15.28:0/2836673328 >> v2:172.21.15.191:3300/0 conn(0x7f1fa80ca520 0x7f1fa80cb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.811+0000 7f1fb0800640 1 --2- 172.21.15.28:0/2836673328 >> v2:172.21.15.28:3300/0 conn(0x7f1fa80ca120 0x7f1fa80cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.811+0000 7f1fb0800640 1 --2- 172.21.15.28:0/2836673328 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1fa8152da0 0x7f1fa815c130 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.811+0000 7f1fb0800640 1 -- 172.21.15.28:0/2836673328 >> 172.21.15.28:0/2836673328 conn(0x7f1fa8065e30 msgr2=0x7f1fa8066240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.811+0000 7f1fb0800640 1 -- 172.21.15.28:0/2836673328 shutdown_connections 2024-09-13T16:44:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.811+0000 7f1fb0800640 1 -- 172.21.15.28:0/2836673328 wait complete. 2024-09-13T16:44:06.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.812+0000 7f1fb0800640 1 Processor -- start 2024-09-13T16:44:06.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.812+0000 7f1fb0800640 1 -- start start 2024-09-13T16:44:06.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.813+0000 7f1fb0800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fa80ca120 0x7f1fa80c42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:06.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.813+0000 7f1fb0800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fa80ca520 0x7f1fa81f1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:06.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.813+0000 7f1fb0800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1fa8152da0 0x7f1fa8145e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:06.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.813+0000 7f1fb0800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa80b4010 con 0x7f1fa80ca120 2024-09-13T16:44:06.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.813+0000 7f1fb0800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa80b43f0 con 0x7f1fa80ca520 2024-09-13T16:44:06.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.813+0000 7f1fb0800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa80b4840 con 0x7f1fa8152da0 2024-09-13T16:44:06.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.813+0000 7f1fade00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fa80ca120 0x7f1fa80c42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:06.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.814+0000 7f1fade00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fa80ca120 0x7f1fa80c42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41816/0 (socket says 172.21.15.28:41816) 2024-09-13T16:44:06.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.814+0000 7f1fade00640 1 -- 172.21.15.28:0/1500406464 learned_addr learned my addr 172.21.15.28:0/1500406464 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:06.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.814+0000 7f1fae800640 1 --2- 172.21.15.28:0/1500406464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1fa8152da0 0x7f1fa8145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:06.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.814+0000 7f1fad400640 1 --2- 172.21.15.28:0/1500406464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fa80ca520 0x7f1fa81f1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:06.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.814+0000 7f1fade00640 1 -- 172.21.15.28:0/1500406464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1fa8152da0 msgr2=0x7f1fa8145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:06.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.814+0000 7f1fade00640 1 --2- 172.21.15.28:0/1500406464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1fa8152da0 0x7f1fa8145e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.814+0000 7f1fade00640 1 -- 172.21.15.28:0/1500406464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fa80ca520 msgr2=0x7f1fa81f1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:06.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.814+0000 7f1fade00640 1 --2- 172.21.15.28:0/1500406464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fa80ca520 0x7f1fa81f1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:06.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.814+0000 7f1fade00640 1 -- 172.21.15.28:0/1500406464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9c0094d0 con 0x7f1fa80ca120 2024-09-13T16:44:06.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.814+0000 7f1fade00640 1 --2- 172.21.15.28:0/1500406464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fa80ca120 0x7f1fa80c42b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c02a2b0 tx=0x7f1f9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:06.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.814+0000 7f1fa2000640 1 -- 172.21.15.28:0/1500406464 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c03b840 con 0x7f1fa80ca120 2024-09-13T16:44:06.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.814+0000 7f1fb0800640 1 -- 172.21.15.28:0/1500406464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa80b3ae0 con 0x7f1fa80ca120 2024-09-13T16:44:06.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.814+0000 7f1fa2000640 1 -- 172.21.15.28:0/1500406464 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9c03b9a0 con 0x7f1fa80ca120 2024-09-13T16:44:06.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.814+0000 7f1fa2000640 1 -- 172.21.15.28:0/1500406464 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c03bd10 con 0x7f1fa80ca120 2024-09-13T16:44:06.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.815+0000 7f1fb0800640 1 -- 172.21.15.28:0/1500406464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa8148660 con 0x7f1fa80ca120 2024-09-13T16:44:06.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.816+0000 7f1fb0800640 1 -- 172.21.15.28:0/1500406464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa80cd920 con 0x7f1fa80ca120 2024-09-13T16:44:06.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.816+0000 7f1fa2000640 1 -- 172.21.15.28:0/1500406464 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1f9c039040 con 0x7f1fa80ca120 2024-09-13T16:44:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.817+0000 7f1fa2000640 1 --2- 172.21.15.28:0/1500406464 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1f8c06ac00 0x7f1f8c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.817+0000 7f1fa2000640 1 -- 172.21.15.28:0/1500406464 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f9c0a9d80 con 0x7f1fa80ca120 2024-09-13T16:44:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.818+0000 7f1fad400640 1 --2- 172.21.15.28:0/1500406464 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1f8c06ac00 0x7f1f8c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.818+0000 7f1fad400640 1 --2- 172.21.15.28:0/1500406464 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1f8c06ac00 0x7f1f8c06d0c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1fa815c5a0 tx=0x7f1f98008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:06.821+0000 7f1fa2000640 1 -- 172.21.15.28:0/1500406464 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1f9c074e40 con 0x7f1fa80ca120 2024-09-13T16:44:07.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:07.039+0000 7f1fb0800640 1 -- 172.21.15.28:0/1500406464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1fa815c130 con 0x7f1fa80ca120 2024-09-13T16:44:07.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:07.039+0000 7f1fa2000640 1 -- 172.21.15.28:0/1500406464 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f1f9c078460 con 0x7f1fa80ca120 2024-09-13T16:44:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:07.041 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-09-13T16:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:07.041+0000 7f1fb0800640 1 -- 172.21.15.28:0/1500406464 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1f8c06ac00 msgr2=0x7f1f8c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:07.041+0000 7f1fb0800640 1 --2- 172.21.15.28:0/1500406464 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1f8c06ac00 0x7f1f8c06d0c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1fa815c5a0 tx=0x7f1f98008040 comp rx=0 tx=0).stop 2024-09-13T16:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:07.041+0000 7f1fb0800640 1 -- 172.21.15.28:0/1500406464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fa80ca120 msgr2=0x7f1fa80c42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:07.041+0000 7f1fb0800640 1 --2- 172.21.15.28:0/1500406464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fa80ca120 0x7f1fa80c42b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c02a2b0 tx=0x7f1f9c0046c0 comp rx=0 tx=0).stop 2024-09-13T16:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:07.041+0000 7f1fb0800640 1 -- 172.21.15.28:0/1500406464 shutdown_connections 2024-09-13T16:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:07.041+0000 7f1fb0800640 1 --2- 172.21.15.28:0/1500406464 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1f8c06ac00 0x7f1f8c06d0c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:07.041+0000 7f1fb0800640 1 --2- 172.21.15.28:0/1500406464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1fa8152da0 0x7f1fa8145e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:07.041+0000 7f1fb0800640 1 --2- 172.21.15.28:0/1500406464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fa80ca520 0x7f1fa81f1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:07.041+0000 7f1fb0800640 1 --2- 172.21.15.28:0/1500406464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fa80ca120 0x7f1fa80c42b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:07.041+0000 7f1fb0800640 1 -- 172.21.15.28:0/1500406464 >> 172.21.15.28:0/1500406464 conn(0x7f1fa8065e30 msgr2=0x7f1fa81505c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:07.041+0000 7f1fb0800640 1 -- 172.21.15.28:0/1500406464 shutdown_connections 2024-09-13T16:44:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:07.041+0000 7f1fb0800640 1 -- 172.21.15.28:0/1500406464 wait complete. 2024-09-13T16:44:10.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T16:44:10.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.224+0000 7f6b5e800640 1 Processor -- start 2024-09-13T16:44:10.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.224+0000 7f6b5e800640 1 -- start start 2024-09-13T16:44:10.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.225+0000 7f6b5e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6b580c2250 0x7f6b58162230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:10.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.225+0000 7f6b5e800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b58151080 con 0x7f6b58150ca0 2024-09-13T16:44:10.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.226+0000 7f6b5e800640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5806a990 con 0x7f6b581568e0 2024-09-13T16:44:10.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.226+0000 7f6b5e800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b580c2680 con 0x7f6b580c2250 2024-09-13T16:44:10.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.226+0000 7f6b57400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6b58150ca0 0x7f6b5815a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38608/0 (socket says 172.21.15.28:38608) 2024-09-13T16:44:10.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.226+0000 7f6b4ea00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f6b581568e0 0x7f6b580c4d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:52110/0 (socket says 172.21.15.28:52110) 2024-09-13T16:44:10.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.226+0000 7f6b57400640 1 -- 172.21.15.28:0/3224160088 learned_addr learned my addr 172.21.15.28:0/3224160088 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:10.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.226+0000 7f6b57e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6b580c2250 0x7f6b58162230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:10.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.226+0000 7f6b56a00640 1 -- 172.21.15.28:0/3224160088 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627046341 0 0) 0x7f6b5806a990 con 0x7f6b581568e0 2024-09-13T16:44:10.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.226+0000 7f6b56a00640 1 -- 172.21.15.28:0/3224160088 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b3c003520 con 0x7f6b581568e0 2024-09-13T16:44:10.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.226+0000 7f6b57e00640 1 -- 172.21.15.28:0/3224160088 >> v1:172.21.15.191:6789/0 conn(0x7f6b581568e0 legacy=0x7f6b580c4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:10.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.226+0000 7f6b57e00640 1 -- 172.21.15.28:0/3224160088 >> v1:172.21.15.28:6789/0 conn(0x7f6b58150ca0 legacy=0x7f6b5815a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:10.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.226+0000 7f6b57e00640 1 -- 172.21.15.28:0/3224160088 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b580c2870 con 0x7f6b580c2250 2024-09-13T16:44:10.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.227+0000 7f6b57e00640 1 --2- 172.21.15.28:0/3224160088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b580c2250 0x7f6b58162230 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6b50009d90 tx=0x7f6b5002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f673ff0d990f7156 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:10.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.227+0000 7f6b56a00640 1 -- 172.21.15.28:0/3224160088 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b50007740 con 0x7f6b580c2250 2024-09-13T16:44:10.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.228+0000 7f6b56a00640 1 -- 172.21.15.28:0/3224160088 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5002d050 con 0x7f6b580c2250 2024-09-13T16:44:10.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.228+0000 7f6b56a00640 1 -- 172.21.15.28:0/3224160088 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b50007ac0 con 0x7f6b580c2250 2024-09-13T16:44:10.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.228+0000 7f6b5e800640 1 -- 172.21.15.28:0/3224160088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b580c2250 msgr2=0x7f6b58162230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:10.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.228+0000 7f6b5e800640 1 --2- 172.21.15.28:0/3224160088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b580c2250 0x7f6b58162230 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6b50009d90 tx=0x7f6b5002a660 comp rx=0 tx=0).stop 2024-09-13T16:44:10.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.228+0000 7f6b5e800640 1 -- 172.21.15.28:0/3224160088 shutdown_connections 2024-09-13T16:44:10.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.228+0000 7f6b5e800640 1 --2- 172.21.15.28:0/3224160088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b580c2250 0x7f6b58162230 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:10.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.228+0000 7f6b5e800640 1 -- 172.21.15.28:0/3224160088 >> 172.21.15.28:0/3224160088 conn(0x7f6b58066040 msgr2=0x7f6b58069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:10.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.228+0000 7f6b5e800640 1 -- 172.21.15.28:0/3224160088 shutdown_connections 2024-09-13T16:44:10.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.228+0000 7f6b5e800640 1 -- 172.21.15.28:0/3224160088 wait complete. 2024-09-13T16:44:10.234 INFO:tasks.ceph.mds.d.smithi191.stderr:2024-09-13T16:44:10.231+0000 7fc0aef7b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:44:10.234 INFO:tasks.ceph.mds.d.smithi191.stderr:2024-09-13T16:44:10.231+0000 7fc0aef7b640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-09-13T16:44:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.230+0000 7f6b5e800640 1 Processor -- start 2024-09-13T16:44:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.230+0000 7f6b5e800640 1 -- start start 2024-09-13T16:44:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.231+0000 7f6b5e800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b580c2250 0x7f6b580711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.231+0000 7f6b5e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58150ca0 0x7f6b5814ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.231+0000 7f6b5e800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b581568e0 0x7f6b5814f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.231+0000 7f6b5e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5814ea40 con 0x7f6b58150ca0 2024-09-13T16:44:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.231+0000 7f6b5e800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b58150550 con 0x7f6b580c2250 2024-09-13T16:44:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.231+0000 7f6b5e800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5814f9e0 con 0x7f6b581568e0 2024-09-13T16:44:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.231+0000 7f6b57400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b580c2250 0x7f6b580711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.231+0000 7f6b4ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58150ca0 0x7f6b5814ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.231+0000 7f6b57400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b580c2250 0x7f6b580711b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:44624/0 (socket says 172.21.15.28:44624) 2024-09-13T16:44:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.231+0000 7f6b57400640 1 -- 172.21.15.28:0/3319050786 learned_addr learned my addr 172.21.15.28:0/3319050786 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.231+0000 7f6b57e00640 1 --2- 172.21.15.28:0/3319050786 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b581568e0 0x7f6b5814f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.231+0000 7f6b4ea00640 1 -- 172.21.15.28:0/3319050786 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b581568e0 msgr2=0x7f6b5814f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.231+0000 7f6b4ea00640 1 --2- 172.21.15.28:0/3319050786 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b581568e0 0x7f6b5814f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.232+0000 7f6b4ea00640 1 -- 172.21.15.28:0/3319050786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b580c2250 msgr2=0x7f6b580711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.232+0000 7f6b4ea00640 1 --2- 172.21.15.28:0/3319050786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b580c2250 0x7f6b580711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.232+0000 7f6b4ea00640 1 -- 172.21.15.28:0/3319050786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b50009a70 con 0x7f6b58150ca0 2024-09-13T16:44:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.232+0000 7f6b57400640 1 --2- 172.21.15.28:0/3319050786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b580c2250 0x7f6b580711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.232+0000 7f6b4ea00640 1 --2- 172.21.15.28:0/3319050786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58150ca0 0x7f6b5814ee60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000c3c0 tx=0x7f6b40001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.232+0000 7f6b4fe00640 1 -- 172.21.15.28:0/3319050786 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b400074e0 con 0x7f6b58150ca0 2024-09-13T16:44:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.232+0000 7f6b4fe00640 1 -- 172.21.15.28:0/3319050786 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b40007660 con 0x7f6b58150ca0 2024-09-13T16:44:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.232+0000 7f6b5e800640 1 -- 172.21.15.28:0/3319050786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b5814fca0 con 0x7f6b58150ca0 2024-09-13T16:44:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.232+0000 7f6b4fe00640 1 -- 172.21.15.28:0/3319050786 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40004a20 con 0x7f6b58150ca0 2024-09-13T16:44:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.232+0000 7f6b5e800640 1 -- 172.21.15.28:0/3319050786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b58150270 con 0x7f6b58150ca0 2024-09-13T16:44:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.233+0000 7f6b5e800640 1 -- 172.21.15.28:0/3319050786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b58154cd0 con 0x7f6b58150ca0 2024-09-13T16:44:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.234+0000 7f6b4fe00640 1 -- 172.21.15.28:0/3319050786 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6b40004b80 con 0x7f6b58150ca0 2024-09-13T16:44:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.235+0000 7f6b4fe00640 1 --2- 172.21.15.28:0/3319050786 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6b3406ac00 0x7f6b3406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.236+0000 7f6b4fe00640 1 -- 172.21.15.28:0/3319050786 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b40088ca0 con 0x7f6b58150ca0 2024-09-13T16:44:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.236+0000 7f6b57400640 1 --2- 172.21.15.28:0/3319050786 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6b3406ac00 0x7f6b3406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.236+0000 7f6b57400640 1 --2- 172.21.15.28:0/3319050786 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6b3406ac00 0x7f6b3406d0c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6b48005880 tx=0x7f6b48009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:10.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.240+0000 7f6b4fe00640 1 -- 172.21.15.28:0/3319050786 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6b40053e90 con 0x7f6b58150ca0 2024-09-13T16:44:10.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.457+0000 7f6b5e800640 1 -- 172.21.15.28:0/3319050786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6b5815a210 con 0x7f6b58150ca0 2024-09-13T16:44:10.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.457+0000 7f6b4fe00640 1 -- 172.21.15.28:0/3319050786 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f6b400574b0 con 0x7f6b58150ca0 2024-09-13T16:44:10.459 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:10.459 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-09-13T16:44:10.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.459+0000 7f6b5e800640 1 -- 172.21.15.28:0/3319050786 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6b3406ac00 msgr2=0x7f6b3406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:10.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.459+0000 7f6b5e800640 1 --2- 172.21.15.28:0/3319050786 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6b3406ac00 0x7f6b3406d0c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6b48005880 tx=0x7f6b48009040 comp rx=0 tx=0).stop 2024-09-13T16:44:10.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.459+0000 7f6b5e800640 1 -- 172.21.15.28:0/3319050786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58150ca0 msgr2=0x7f6b5814ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:10.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.459+0000 7f6b5e800640 1 --2- 172.21.15.28:0/3319050786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58150ca0 0x7f6b5814ee60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000c3c0 tx=0x7f6b40001e70 comp rx=0 tx=0).stop 2024-09-13T16:44:10.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.459+0000 7f6b5e800640 1 -- 172.21.15.28:0/3319050786 shutdown_connections 2024-09-13T16:44:10.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.459+0000 7f6b5e800640 1 --2- 172.21.15.28:0/3319050786 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6b3406ac00 0x7f6b3406d0c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:10.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.459+0000 7f6b5e800640 1 --2- 172.21.15.28:0/3319050786 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b581568e0 0x7f6b5814f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:10.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.459+0000 7f6b5e800640 1 --2- 172.21.15.28:0/3319050786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b58150ca0 0x7f6b5814ee60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:10.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.459+0000 7f6b5e800640 1 --2- 172.21.15.28:0/3319050786 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b580c2250 0x7f6b580711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:10.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.459+0000 7f6b5e800640 1 -- 172.21.15.28:0/3319050786 >> 172.21.15.28:0/3319050786 conn(0x7f6b58066040 msgr2=0x7f6b580c5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:10.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.460+0000 7f6b5e800640 1 -- 172.21.15.28:0/3319050786 shutdown_connections 2024-09-13T16:44:10.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:10.460+0000 7f6b5e800640 1 -- 172.21.15.28:0/3319050786 wait complete. 2024-09-13T16:44:13.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T16:44:13.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.640+0000 7f2304a00640 1 Processor -- start 2024-09-13T16:44:13.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.640+0000 7f2304a00640 1 -- start start 2024-09-13T16:44:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.641+0000 7f2304a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23000c5bd0 0x7f2300156150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.642+0000 7f2304a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f23000c3670 0x7f23000c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.642+0000 7f2304a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230014ca00 con 0x7f23000c1ea0 2024-09-13T16:44:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.642+0000 7f2304a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23000c2dc0 con 0x7f23000c3670 2024-09-13T16:44:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.642+0000 7f2304a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2300155780 con 0x7f23000c5bd0 2024-09-13T16:44:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.642+0000 7f22f5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f23000c3670 0x7f23000c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.642+0000 7f22f5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f23000c3670 0x7f23000c1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42398/0 (socket says 172.21.15.28:42398) 2024-09-13T16:44:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.642+0000 7f22f5400640 1 -- 172.21.15.28:0/421919636 learned_addr learned my addr 172.21.15.28:0/421919636 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.642+0000 7f22fde00640 1 --2- 172.21.15.28:0/421919636 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23000c5bd0 0x7f2300156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.642+0000 7f22f5400640 1 -- 172.21.15.28:0/421919636 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23000c5bd0 msgr2=0x7f2300156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.642+0000 7f22f5400640 1 --2- 172.21.15.28:0/421919636 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23000c5bd0 0x7f2300156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:13.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.642+0000 7f22f5400640 1 -- 172.21.15.28:0/421919636 >> v1:172.21.15.28:6789/0 conn(0x7f23000c1ea0 legacy=0x7f23000c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:13.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.642+0000 7f22f5400640 1 -- 172.21.15.28:0/421919636 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2300155990 con 0x7f23000c3670 2024-09-13T16:44:13.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.642+0000 7f22fd400640 1 -- 172.21.15.28:0/421919636 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277622997 0 0) 0x7f230014ca00 con 0x7f23000c1ea0 2024-09-13T16:44:13.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.643+0000 7f22f5400640 1 --2- 172.21.15.28:0/421919636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23000c3670 0x7f23000c1960 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f22e800bb10 tx=0x7f22e802c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35737ace0b39a3ac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:13.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.644+0000 7f22fd400640 1 -- 172.21.15.28:0/421919636 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8031610 con 0x7f23000c3670 2024-09-13T16:44:13.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.644+0000 7f22fd400640 1 -- 172.21.15.28:0/421919636 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e8031770 con 0x7f23000c3670 2024-09-13T16:44:13.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.644+0000 7f22fd400640 1 -- 172.21.15.28:0/421919636 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8031b00 con 0x7f23000c3670 2024-09-13T16:44:13.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.644+0000 7f2304a00640 1 -- 172.21.15.28:0/421919636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23000c3670 msgr2=0x7f23000c1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:13.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.644+0000 7f2304a00640 1 --2- 172.21.15.28:0/421919636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23000c3670 0x7f23000c1960 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f22e800bb10 tx=0x7f22e802c9b0 comp rx=0 tx=0).stop 2024-09-13T16:44:13.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.644+0000 7f2304a00640 1 -- 172.21.15.28:0/421919636 shutdown_connections 2024-09-13T16:44:13.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.644+0000 7f2304a00640 1 --2- 172.21.15.28:0/421919636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23000c3670 0x7f23000c1960 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:13.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.644+0000 7f2304a00640 1 --2- 172.21.15.28:0/421919636 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23000c5bd0 0x7f2300156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:13.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.644+0000 7f2304a00640 1 -- 172.21.15.28:0/421919636 >> 172.21.15.28:0/421919636 conn(0x7f2300066040 msgr2=0x7f2300069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:13.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.644+0000 7f2304a00640 1 -- 172.21.15.28:0/421919636 shutdown_connections 2024-09-13T16:44:13.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.644+0000 7f2304a00640 1 -- 172.21.15.28:0/421919636 wait complete. 2024-09-13T16:44:13.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.646+0000 7f2304a00640 1 Processor -- start 2024-09-13T16:44:13.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.646+0000 7f2304a00640 1 -- start start 2024-09-13T16:44:13.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.647+0000 7f2304a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23000c1ea0 0x7f2300144d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:13.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.647+0000 7f2304a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23000c3670 0x7f23001edc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:13.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.647+0000 7f2304a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23000c5bd0 0x7f23001ee190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:13.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.647+0000 7f2304a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2300146580 con 0x7f23000c5bd0 2024-09-13T16:44:13.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.647+0000 7f2304a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23001ed800 con 0x7f23000c3670 2024-09-13T16:44:13.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.647+0000 7f2304a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2300145720 con 0x7f23000c1ea0 2024-09-13T16:44:13.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.647+0000 7f22f5400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23000c3670 0x7f23001edc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:13.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.647+0000 7f22fde00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23000c1ea0 0x7f2300144d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:13.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.647+0000 7f22fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23000c5bd0 0x7f23001ee190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:13.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.647+0000 7f22fde00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23000c1ea0 0x7f2300144d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:33220/0 (socket says 172.21.15.28:33220) 2024-09-13T16:44:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.647+0000 7f22fde00640 1 -- 172.21.15.28:0/3650170816 learned_addr learned my addr 172.21.15.28:0/3650170816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.647+0000 7f22f5400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23000c3670 0x7f23001edc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:44634/0 (socket says 172.21.15.28:44634) 2024-09-13T16:44:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.648+0000 7f22fde00640 1 -- 172.21.15.28:0/3650170816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23000c3670 msgr2=0x7f23001edc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.648+0000 7f22fde00640 1 --2- 172.21.15.28:0/3650170816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23000c3670 0x7f23001edc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.648+0000 7f22fde00640 1 -- 172.21.15.28:0/3650170816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23000c5bd0 msgr2=0x7f23001ee190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.648+0000 7f22fde00640 1 --2- 172.21.15.28:0/3650170816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23000c5bd0 0x7f23001ee190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.648+0000 7f22fde00640 1 -- 172.21.15.28:0/3650170816 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e800b770 con 0x7f23000c1ea0 2024-09-13T16:44:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.648+0000 7f22fde00640 1 --2- 172.21.15.28:0/3650170816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23000c1ea0 0x7f2300144d20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f22f000e870 tx=0x7f22f000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:13.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.649+0000 7f22f6a00640 1 -- 172.21.15.28:0/3650170816 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f00094f0 con 0x7f23000c1ea0 2024-09-13T16:44:13.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.649+0000 7f22f6a00640 1 -- 172.21.15.28:0/3650170816 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f000c940 con 0x7f23000c1ea0 2024-09-13T16:44:13.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.649+0000 7f2304a00640 1 -- 172.21.15.28:0/3650170816 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2300145a00 con 0x7f23000c1ea0 2024-09-13T16:44:13.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.649+0000 7f22f6a00640 1 -- 172.21.15.28:0/3650170816 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f000ccf0 con 0x7f23000c1ea0 2024-09-13T16:44:13.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.649+0000 7f2304a00640 1 -- 172.21.15.28:0/3650170816 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2300145fc0 con 0x7f23000c1ea0 2024-09-13T16:44:13.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.650+0000 7f2304a00640 1 -- 172.21.15.28:0/3650170816 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23001ed970 con 0x7f23000c1ea0 2024-09-13T16:44:13.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.651+0000 7f22f6a00640 1 -- 172.21.15.28:0/3650170816 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f22f0013470 con 0x7f23000c1ea0 2024-09-13T16:44:13.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.652+0000 7f22f6a00640 1 --2- 172.21.15.28:0/3650170816 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f22dc06ac00 0x7f22dc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:13.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.652+0000 7f22f6a00640 1 -- 172.21.15.28:0/3650170816 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f22f00894c0 con 0x7f23000c1ea0 2024-09-13T16:44:13.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.652+0000 7f22f5400640 1 --2- 172.21.15.28:0/3650170816 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f22dc06ac00 0x7f22dc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:13.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.653+0000 7f22f5400640 1 --2- 172.21.15.28:0/3650170816 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f22dc06ac00 0x7f22dc06d0c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f22e8037000 tx=0x7f22e8032000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.656+0000 7f22f6a00640 1 -- 172.21.15.28:0/3650170816 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f22f0054620 con 0x7f23000c1ea0 2024-09-13T16:44:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.872+0000 7f2304a00640 1 -- 172.21.15.28:0/3650170816 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f23000c8b70 con 0x7f23000c1ea0 2024-09-13T16:44:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.872+0000 7f22f6a00640 1 -- 172.21.15.28:0/3650170816 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f23000c8b70 con 0x7f23000c1ea0 2024-09-13T16:44:13.874 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:13.874 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-09-13T16:44:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.875+0000 7f2304a00640 1 -- 172.21.15.28:0/3650170816 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f22dc06ac00 msgr2=0x7f22dc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.875+0000 7f2304a00640 1 --2- 172.21.15.28:0/3650170816 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f22dc06ac00 0x7f22dc06d0c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f22e8037000 tx=0x7f22e8032000 comp rx=0 tx=0).stop 2024-09-13T16:44:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.875+0000 7f2304a00640 1 -- 172.21.15.28:0/3650170816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23000c1ea0 msgr2=0x7f2300144d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.875+0000 7f2304a00640 1 --2- 172.21.15.28:0/3650170816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23000c1ea0 0x7f2300144d20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f22f000e870 tx=0x7f22f000ed30 comp rx=0 tx=0).stop 2024-09-13T16:44:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.875+0000 7f2304a00640 1 -- 172.21.15.28:0/3650170816 shutdown_connections 2024-09-13T16:44:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.875+0000 7f2304a00640 1 --2- 172.21.15.28:0/3650170816 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f22dc06ac00 0x7f22dc06d0c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.875+0000 7f2304a00640 1 --2- 172.21.15.28:0/3650170816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23000c5bd0 0x7f23001ee190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.875+0000 7f2304a00640 1 --2- 172.21.15.28:0/3650170816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23000c3670 0x7f23001edc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.875+0000 7f2304a00640 1 --2- 172.21.15.28:0/3650170816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23000c1ea0 0x7f2300144d20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.875+0000 7f2304a00640 1 -- 172.21.15.28:0/3650170816 >> 172.21.15.28:0/3650170816 conn(0x7f2300066040 msgr2=0x7f23000c6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.875+0000 7f2304a00640 1 -- 172.21.15.28:0/3650170816 shutdown_connections 2024-09-13T16:44:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:13.875+0000 7f2304a00640 1 -- 172.21.15.28:0/3650170816 wait complete. 2024-09-13T16:44:15.233 INFO:tasks.ceph.mds.d.smithi191.stderr:2024-09-13T16:44:15.230+0000 7fc0aef7b640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-09-13T16:44:16.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T16:44:17.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.056+0000 7f3cabe00640 1 Processor -- start 2024-09-13T16:44:17.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.056+0000 7f3cabe00640 1 -- start start 2024-09-13T16:44:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.057+0000 7f3cabe00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3ca4150ca0 0x7f3ca41568e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.057+0000 7f3cabe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ca40c2250 0x7f3ca415a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.057+0000 7f3cabe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca4151080 con 0x7f3ca40c4d60 2024-09-13T16:44:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.058+0000 7f3cabe00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ca40c2670 con 0x7f3ca4150ca0 2024-09-13T16:44:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.058+0000 7f3cabe00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca406a990 con 0x7f3ca40c2250 2024-09-13T16:44:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.058+0000 7f3ca9400640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3ca4150ca0 0x7f3ca41568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.058+0000 7f3ca8a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ca40c2250 0x7f3ca415a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.058+0000 7f3ca9400640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3ca4150ca0 0x7f3ca41568e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:44644/0 (socket says 172.21.15.28:44644) 2024-09-13T16:44:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.058+0000 7f3ca9400640 1 -- 172.21.15.28:0/1703210762 learned_addr learned my addr 172.21.15.28:0/1703210762 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.058+0000 7f3ca9e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3ca40c4d60 0x7f3ca406d8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38622/0 (socket says 172.21.15.28:38622) 2024-09-13T16:44:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.058+0000 7f3ca8a00640 1 -- 172.21.15.28:0/1703210762 >> v2:172.21.15.191:3300/0 conn(0x7f3ca4150ca0 msgr2=0x7f3ca41568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.058+0000 7f3ca8a00640 1 --2- 172.21.15.28:0/1703210762 >> v2:172.21.15.191:3300/0 conn(0x7f3ca4150ca0 0x7f3ca41568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.058+0000 7f3ca8a00640 1 -- 172.21.15.28:0/1703210762 >> v1:172.21.15.28:6789/0 conn(0x7f3ca40c4d60 legacy=0x7f3ca406d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.058+0000 7f3ca8a00640 1 -- 172.21.15.28:0/1703210762 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca40c2870 con 0x7f3ca40c2250 2024-09-13T16:44:17.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.058+0000 7f3ca3e00640 1 -- 172.21.15.28:0/1703210762 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157299399 0 0) 0x7f3ca4151080 con 0x7f3ca40c4d60 2024-09-13T16:44:17.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.058+0000 7f3ca9400640 1 --2- 172.21.15.28:0/1703210762 >> v2:172.21.15.191:3300/0 conn(0x7f3ca4150ca0 0x7f3ca41568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:17.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.059+0000 7f3ca8a00640 1 --2- 172.21.15.28:0/1703210762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ca40c2250 0x7f3ca415a730 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3c900097a0 tx=0x7f3c9002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d9462925b57d29e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:17.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.060+0000 7f3ca3e00640 1 -- 172.21.15.28:0/1703210762 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9002ae50 con 0x7f3ca40c2250 2024-09-13T16:44:17.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.060+0000 7f3ca3e00640 1 -- 172.21.15.28:0/1703210762 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c90032440 con 0x7f3ca40c2250 2024-09-13T16:44:17.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.060+0000 7f3ca3e00640 1 -- 172.21.15.28:0/1703210762 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c900327d0 con 0x7f3ca40c2250 2024-09-13T16:44:17.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.060+0000 7f3cabe00640 1 -- 172.21.15.28:0/1703210762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ca40c2250 msgr2=0x7f3ca415a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:17.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.060+0000 7f3cabe00640 1 --2- 172.21.15.28:0/1703210762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ca40c2250 0x7f3ca415a730 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3c900097a0 tx=0x7f3c9002a100 comp rx=0 tx=0).stop 2024-09-13T16:44:17.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.060+0000 7f3cabe00640 1 -- 172.21.15.28:0/1703210762 shutdown_connections 2024-09-13T16:44:17.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.060+0000 7f3cabe00640 1 --2- 172.21.15.28:0/1703210762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ca40c2250 0x7f3ca415a730 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:17.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.060+0000 7f3cabe00640 1 --2- 172.21.15.28:0/1703210762 >> v2:172.21.15.191:3300/0 conn(0x7f3ca4150ca0 0x7f3ca41568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:17.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.060+0000 7f3cabe00640 1 -- 172.21.15.28:0/1703210762 >> 172.21.15.28:0/1703210762 conn(0x7f3ca4066040 msgr2=0x7f3ca4069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:17.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.060+0000 7f3cabe00640 1 -- 172.21.15.28:0/1703210762 shutdown_connections 2024-09-13T16:44:17.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.060+0000 7f3cabe00640 1 -- 172.21.15.28:0/1703210762 wait complete. 2024-09-13T16:44:17.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.062+0000 7f3cabe00640 1 Processor -- start 2024-09-13T16:44:17.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.062+0000 7f3cabe00640 1 -- start start 2024-09-13T16:44:17.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.063+0000 7f3cabe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ca40c2250 0x7f3ca41f2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:17.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.063+0000 7f3cabe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ca40c4d60 0x7f3ca41eec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.063+0000 7f3cabe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ca4150ca0 0x7f3ca4155b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.063+0000 7f3cabe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca4158b20 con 0x7f3ca40c2250 2024-09-13T16:44:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.063+0000 7f3cabe00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca41f2840 con 0x7f3ca40c4d60 2024-09-13T16:44:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.063+0000 7f3cabe00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca4158d50 con 0x7f3ca4150ca0 2024-09-13T16:44:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.063+0000 7f3ca9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ca40c2250 0x7f3ca41f2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.063+0000 7f3ca9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ca40c2250 0x7f3ca41f2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42418/0 (socket says 172.21.15.28:42418) 2024-09-13T16:44:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.063+0000 7f3ca9400640 1 -- 172.21.15.28:0/639565303 learned_addr learned my addr 172.21.15.28:0/639565303 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.063+0000 7f3ca9e00640 1 --2- 172.21.15.28:0/639565303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ca4150ca0 0x7f3ca4155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.063+0000 7f3ca8a00640 1 --2- 172.21.15.28:0/639565303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ca40c4d60 0x7f3ca41eec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.063+0000 7f3ca9400640 1 -- 172.21.15.28:0/639565303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ca4150ca0 msgr2=0x7f3ca4155b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.063+0000 7f3ca9400640 1 --2- 172.21.15.28:0/639565303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ca4150ca0 0x7f3ca4155b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.063+0000 7f3ca9400640 1 -- 172.21.15.28:0/639565303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ca40c4d60 msgr2=0x7f3ca41eec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.063+0000 7f3ca9400640 1 --2- 172.21.15.28:0/639565303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ca40c4d60 0x7f3ca41eec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.063+0000 7f3ca9400640 1 -- 172.21.15.28:0/639565303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c980094d0 con 0x7f3ca40c2250 2024-09-13T16:44:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.064+0000 7f3ca8a00640 1 --2- 172.21.15.28:0/639565303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ca40c4d60 0x7f3ca41eec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.064+0000 7f3ca9400640 1 --2- 172.21.15.28:0/639565303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ca40c2250 0x7f3ca41f2300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3c9800e9b0 tx=0x7f3c9800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.064+0000 7f3ca1600640 1 -- 172.21.15.28:0/639565303 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9800c9a0 con 0x7f3ca40c2250 2024-09-13T16:44:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.064+0000 7f3ca1600640 1 -- 172.21.15.28:0/639565303 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9800cb00 con 0x7f3ca40c2250 2024-09-13T16:44:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.064+0000 7f3cabe00640 1 -- 172.21.15.28:0/639565303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c90009400 con 0x7f3ca40c2250 2024-09-13T16:44:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.064+0000 7f3ca1600640 1 -- 172.21.15.28:0/639565303 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9800ce70 con 0x7f3ca40c2250 2024-09-13T16:44:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.064+0000 7f3cabe00640 1 -- 172.21.15.28:0/639565303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca4159310 con 0x7f3ca40c2250 2024-09-13T16:44:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.065+0000 7f3cabe00640 1 -- 172.21.15.28:0/639565303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca40c3110 con 0x7f3ca40c2250 2024-09-13T16:44:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.066+0000 7f3ca1600640 1 -- 172.21.15.28:0/639565303 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3c98013470 con 0x7f3ca40c2250 2024-09-13T16:44:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.067+0000 7f3ca1600640 1 --2- 172.21.15.28:0/639565303 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3c8406ac00 0x7f3c8406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.068+0000 7f3ca1600640 1 -- 172.21.15.28:0/639565303 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c98089650 con 0x7f3ca40c2250 2024-09-13T16:44:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.068+0000 7f3ca8a00640 1 --2- 172.21.15.28:0/639565303 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3c8406ac00 0x7f3c8406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.068+0000 7f3ca8a00640 1 --2- 172.21.15.28:0/639565303 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3c8406ac00 0x7f3c8406d0c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3c90034000 tx=0x7f3c9002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:17.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.072+0000 7f3ca1600640 1 -- 172.21.15.28:0/639565303 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3c980547b0 con 0x7f3ca40c2250 2024-09-13T16:44:17.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.286+0000 7f3cabe00640 1 -- 172.21.15.28:0/639565303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3ca40c5910 con 0x7f3ca40c2250 2024-09-13T16:44:17.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.286+0000 7f3ca1600640 1 -- 172.21.15.28:0/639565303 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f3c98057dd0 con 0x7f3ca40c2250 2024-09-13T16:44:17.288 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:17.288 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-09-13T16:44:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.288+0000 7f3cabe00640 1 -- 172.21.15.28:0/639565303 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3c8406ac00 msgr2=0x7f3c8406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.288+0000 7f3cabe00640 1 --2- 172.21.15.28:0/639565303 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3c8406ac00 0x7f3c8406d0c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3c90034000 tx=0x7f3c9002e000 comp rx=0 tx=0).stop 2024-09-13T16:44:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.288+0000 7f3cabe00640 1 -- 172.21.15.28:0/639565303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ca40c2250 msgr2=0x7f3ca41f2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.288+0000 7f3cabe00640 1 --2- 172.21.15.28:0/639565303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ca40c2250 0x7f3ca41f2300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3c9800e9b0 tx=0x7f3c9800ee70 comp rx=0 tx=0).stop 2024-09-13T16:44:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.288+0000 7f3cabe00640 1 -- 172.21.15.28:0/639565303 shutdown_connections 2024-09-13T16:44:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.288+0000 7f3cabe00640 1 --2- 172.21.15.28:0/639565303 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3c8406ac00 0x7f3c8406d0c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.288+0000 7f3cabe00640 1 --2- 172.21.15.28:0/639565303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ca4150ca0 0x7f3ca4155b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.288+0000 7f3cabe00640 1 --2- 172.21.15.28:0/639565303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ca40c4d60 0x7f3ca41eec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.288+0000 7f3cabe00640 1 --2- 172.21.15.28:0/639565303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ca40c2250 0x7f3ca41f2300 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.288+0000 7f3cabe00640 1 -- 172.21.15.28:0/639565303 >> 172.21.15.28:0/639565303 conn(0x7f3ca4066040 msgr2=0x7f3ca414d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.288+0000 7f3cabe00640 1 -- 172.21.15.28:0/639565303 shutdown_connections 2024-09-13T16:44:17.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:17.288+0000 7f3cabe00640 1 -- 172.21.15.28:0/639565303 wait complete. 2024-09-13T16:44:20.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T16:44:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.468+0000 7fbdf4e00640 1 Processor -- start 2024-09-13T16:44:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.468+0000 7fbdf4e00640 1 -- start start 2024-09-13T16:44:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.470+0000 7fbdf4e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbdf0150ca0 0x7fbdf01568e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.470+0000 7fbdf4e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf00c4d60 0x7fbdf00c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.470+0000 7fbdf4e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf0151080 con 0x7fbdf00c2250 2024-09-13T16:44:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.470+0000 7fbdf4e00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf00c2670 con 0x7fbdf0150ca0 2024-09-13T16:44:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.470+0000 7fbdf4e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf015ae40 con 0x7fbdf00c4d60 2024-09-13T16:44:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.470+0000 7fbde5400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbdf00c2250 0x7fbdf015a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60950/0 (socket says 172.21.15.28:60950) 2024-09-13T16:44:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.470+0000 7fbde5400640 1 -- 172.21.15.28:0/2549075514 learned_addr learned my addr 172.21.15.28:0/2549075514 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.470+0000 7fbdede00640 1 --2- 172.21.15.28:0/2549075514 >> v2:172.21.15.191:3300/0 conn(0x7fbdf0150ca0 0x7fbdf01568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.470+0000 7fbdee800640 1 --2- 172.21.15.28:0/2549075514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf00c4d60 0x7fbdf00c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.470+0000 7fbded400640 1 -- 172.21.15.28:0/2549075514 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799216400 0 0) 0x7fbdf0151080 con 0x7fbdf00c2250 2024-09-13T16:44:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.470+0000 7fbdede00640 1 -- 172.21.15.28:0/2549075514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf00c4d60 msgr2=0x7fbdf00c51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.470+0000 7fbdede00640 1 --2- 172.21.15.28:0/2549075514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf00c4d60 0x7fbdf00c51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.470+0000 7fbdede00640 1 -- 172.21.15.28:0/2549075514 >> v1:172.21.15.28:6789/0 conn(0x7fbdf00c2250 legacy=0x7fbdf015a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.470+0000 7fbdede00640 1 -- 172.21.15.28:0/2549075514 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf006a990 con 0x7fbdf0150ca0 2024-09-13T16:44:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.470+0000 7fbdee800640 1 --2- 172.21.15.28:0/2549075514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf00c4d60 0x7fbdf00c51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:20.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.471+0000 7fbdede00640 1 --2- 172.21.15.28:0/2549075514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdf0150ca0 0x7fbdf01568e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbde00097d0 tx=0x7fbde002a120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acaf718d31d4b814 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:20.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.472+0000 7fbded400640 1 -- 172.21.15.28:0/2549075514 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde002ae30 con 0x7fbdf0150ca0 2024-09-13T16:44:20.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.472+0000 7fbded400640 1 -- 172.21.15.28:0/2549075514 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde0032440 con 0x7fbdf0150ca0 2024-09-13T16:44:20.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.472+0000 7fbded400640 1 -- 172.21.15.28:0/2549075514 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde00327d0 con 0x7fbdf0150ca0 2024-09-13T16:44:20.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.472+0000 7fbdf4e00640 1 -- 172.21.15.28:0/2549075514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdf0150ca0 msgr2=0x7fbdf01568e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.472+0000 7fbdf4e00640 1 --2- 172.21.15.28:0/2549075514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdf0150ca0 0x7fbdf01568e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbde00097d0 tx=0x7fbde002a120 comp rx=0 tx=0).stop 2024-09-13T16:44:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.472+0000 7fbdf4e00640 1 -- 172.21.15.28:0/2549075514 shutdown_connections 2024-09-13T16:44:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.472+0000 7fbdf4e00640 1 --2- 172.21.15.28:0/2549075514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf00c4d60 0x7fbdf00c51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.472+0000 7fbdf4e00640 1 --2- 172.21.15.28:0/2549075514 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdf0150ca0 0x7fbdf01568e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.472+0000 7fbdf4e00640 1 -- 172.21.15.28:0/2549075514 >> 172.21.15.28:0/2549075514 conn(0x7fbdf0066040 msgr2=0x7fbdf0069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.472+0000 7fbdf4e00640 1 -- 172.21.15.28:0/2549075514 shutdown_connections 2024-09-13T16:44:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.473+0000 7fbdf4e00640 1 -- 172.21.15.28:0/2549075514 wait complete. 2024-09-13T16:44:20.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.474+0000 7fbdf4e00640 1 Processor -- start 2024-09-13T16:44:20.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.474+0000 7fbdf4e00640 1 -- start start 2024-09-13T16:44:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.475+0000 7fbdf4e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdf00c2250 0x7fbdf0158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.475+0000 7fbdf4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdf00c4d60 0x7fbdf0142f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.475+0000 7fbdf4e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf0150ca0 0x7fbdf0159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.475+0000 7fbdf4e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf0156790 con 0x7fbdf00c4d60 2024-09-13T16:44:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.475+0000 7fbdf4e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf0158960 con 0x7fbdf00c2250 2024-09-13T16:44:20.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.475+0000 7fbdf4e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf0158fb0 con 0x7fbdf0150ca0 2024-09-13T16:44:20.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.475+0000 7fbde5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdf00c4d60 0x7fbdf0142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:20.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.475+0000 7fbdede00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdf00c2250 0x7fbdf0158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:20.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.475+0000 7fbdede00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdf00c2250 0x7fbdf0158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:50936/0 (socket says 172.21.15.28:50936) 2024-09-13T16:44:20.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.475+0000 7fbdede00640 1 -- 172.21.15.28:0/3744742018 learned_addr learned my addr 172.21.15.28:0/3744742018 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:20.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.475+0000 7fbdee800640 1 --2- 172.21.15.28:0/3744742018 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf0150ca0 0x7fbdf0159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:20.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.476+0000 7fbdede00640 1 -- 172.21.15.28:0/3744742018 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf0150ca0 msgr2=0x7fbdf0159460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:20.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.476+0000 7fbdede00640 1 --2- 172.21.15.28:0/3744742018 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf0150ca0 0x7fbdf0159460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:20.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.476+0000 7fbdede00640 1 -- 172.21.15.28:0/3744742018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdf00c4d60 msgr2=0x7fbdf0142f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:20.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.476+0000 7fbdede00640 1 --2- 172.21.15.28:0/3744742018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdf00c4d60 0x7fbdf0142f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:20.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.476+0000 7fbdede00640 1 -- 172.21.15.28:0/3744742018 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde8009a70 con 0x7fbdf00c2250 2024-09-13T16:44:20.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.476+0000 7fbde5400640 1 --2- 172.21.15.28:0/3744742018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdf00c4d60 0x7fbdf0142f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:20.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.476+0000 7fbdede00640 1 --2- 172.21.15.28:0/3744742018 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdf00c2250 0x7fbdf0158420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009720 tx=0x7fbde00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.476+0000 7fbde6a00640 1 -- 172.21.15.28:0/3744742018 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0004230 con 0x7fbdf00c2250 2024-09-13T16:44:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.476+0000 7fbde6a00640 1 -- 172.21.15.28:0/3744742018 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde0004390 con 0x7fbdf00c2250 2024-09-13T16:44:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.476+0000 7fbde6a00640 1 -- 172.21.15.28:0/3744742018 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde003c820 con 0x7fbdf00c2250 2024-09-13T16:44:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.476+0000 7fbdf4e00640 1 -- 172.21.15.28:0/3744742018 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde0009400 con 0x7fbdf00c2250 2024-09-13T16:44:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.476+0000 7fbdf4e00640 1 -- 172.21.15.28:0/3744742018 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf014dd60 con 0x7fbdf00c2250 2024-09-13T16:44:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.477+0000 7fbdf4e00640 1 -- 172.21.15.28:0/3744742018 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf0159150 con 0x7fbdf00c2250 2024-09-13T16:44:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.478+0000 7fbde6a00640 1 -- 172.21.15.28:0/3744742018 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbde0039050 con 0x7fbdf00c2250 2024-09-13T16:44:20.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.479+0000 7fbde6a00640 1 --2- 172.21.15.28:0/3744742018 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fbdcc06ac00 0x7fbdcc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:20.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.479+0000 7fbde6a00640 1 -- 172.21.15.28:0/3744742018 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbde00a9d00 con 0x7fbdf00c2250 2024-09-13T16:44:20.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.480+0000 7fbde5400640 1 --2- 172.21.15.28:0/3744742018 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fbdcc06ac00 0x7fbdcc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:20.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.480+0000 7fbde5400640 1 --2- 172.21.15.28:0/3744742018 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fbdcc06ac00 0x7fbdcc06d0c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbdf014edd0 tx=0x7fbdd8007d50 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:20.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.483+0000 7fbde6a00640 1 -- 172.21.15.28:0/3744742018 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbde0075000 con 0x7fbdf00c2250 2024-09-13T16:44:20.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.700+0000 7fbdf4e00640 1 -- 172.21.15.28:0/3744742018 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbdf0159ec0 con 0x7fbdf00c2250 2024-09-13T16:44:20.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.701+0000 7fbde6a00640 1 -- 172.21.15.28:0/3744742018 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fbde0037080 con 0x7fbdf00c2250 2024-09-13T16:44:20.702 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:20.703 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T16:44:20.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.703+0000 7fbdf4e00640 1 -- 172.21.15.28:0/3744742018 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fbdcc06ac00 msgr2=0x7fbdcc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:20.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.703+0000 7fbdf4e00640 1 --2- 172.21.15.28:0/3744742018 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fbdcc06ac00 0x7fbdcc06d0c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbdf014edd0 tx=0x7fbdd8007d50 comp rx=0 tx=0).stop 2024-09-13T16:44:20.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.703+0000 7fbdf4e00640 1 -- 172.21.15.28:0/3744742018 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdf00c2250 msgr2=0x7fbdf0158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:20.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.703+0000 7fbdf4e00640 1 --2- 172.21.15.28:0/3744742018 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdf00c2250 0x7fbdf0158420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009720 tx=0x7fbde00046c0 comp rx=0 tx=0).stop 2024-09-13T16:44:20.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.703+0000 7fbdf4e00640 1 -- 172.21.15.28:0/3744742018 shutdown_connections 2024-09-13T16:44:20.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.703+0000 7fbdf4e00640 1 --2- 172.21.15.28:0/3744742018 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fbdcc06ac00 0x7fbdcc06d0c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:20.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.703+0000 7fbdf4e00640 1 --2- 172.21.15.28:0/3744742018 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdf0150ca0 0x7fbdf0159460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:20.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.703+0000 7fbdf4e00640 1 --2- 172.21.15.28:0/3744742018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdf00c4d60 0x7fbdf0142f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:20.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.703+0000 7fbdf4e00640 1 --2- 172.21.15.28:0/3744742018 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdf00c2250 0x7fbdf0158420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:20.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.703+0000 7fbdf4e00640 1 -- 172.21.15.28:0/3744742018 >> 172.21.15.28:0/3744742018 conn(0x7fbdf0066040 msgr2=0x7fbdf01dfcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:20.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.703+0000 7fbdf4e00640 1 -- 172.21.15.28:0/3744742018 shutdown_connections 2024-09-13T16:44:20.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.703+0000 7fbdf4e00640 1 -- 172.21.15.28:0/3744742018 wait complete. 2024-09-13T16:44:20.720 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-13T16:44:20.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:44:20.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.874+0000 7f7cba600640 1 Processor -- start 2024-09-13T16:44:20.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.874+0000 7f7cba600640 1 -- start start 2024-09-13T16:44:20.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.876+0000 7f7cba600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7cb41568e0 0x7f7cb40c2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:20.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.876+0000 7f7cba600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7cb40c4d60 0x7f7cb40c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:20.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.876+0000 7f7cba600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb4151080 con 0x7f7cb4150ca0 2024-09-13T16:44:20.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.876+0000 7f7cba600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cb415ae40 con 0x7f7cb41568e0 2024-09-13T16:44:20.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.876+0000 7f7cba600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb406a990 con 0x7f7cb40c4d60 2024-09-13T16:44:20.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.876+0000 7f7cb3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7cb4150ca0 0x7f7cb415a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60960/0 (socket says 172.21.15.28:60960) 2024-09-13T16:44:20.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.876+0000 7f7cb3400640 1 -- 172.21.15.28:0/2848102860 learned_addr learned my addr 172.21.15.28:0/2848102860 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:20.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.876+0000 7f7caaa00640 1 --2- 172.21.15.28:0/2848102860 >> v2:172.21.15.28:3300/0 conn(0x7f7cb41568e0 0x7f7cb40c2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:20.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.876+0000 7f7cb3e00640 1 --2- 172.21.15.28:0/2848102860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7cb40c4d60 0x7f7cb40c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:20.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.876+0000 7f7cb2a00640 1 -- 172.21.15.28:0/2848102860 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189300864 0 0) 0x7f7cb4151080 con 0x7f7cb4150ca0 2024-09-13T16:44:20.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.876+0000 7f7cb2a00640 1 -- 172.21.15.28:0/2848102860 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c98003520 con 0x7f7cb4150ca0 2024-09-13T16:44:20.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.876+0000 7f7caaa00640 1 -- 172.21.15.28:0/2848102860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7cb40c4d60 msgr2=0x7f7cb40c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:20.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.876+0000 7f7caaa00640 1 --2- 172.21.15.28:0/2848102860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7cb40c4d60 0x7f7cb40c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:20.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.876+0000 7f7caaa00640 1 -- 172.21.15.28:0/2848102860 >> v1:172.21.15.28:6789/0 conn(0x7f7cb4150ca0 legacy=0x7f7cb415a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:20.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.876+0000 7f7caaa00640 1 -- 172.21.15.28:0/2848102860 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb40c2870 con 0x7f7cb41568e0 2024-09-13T16:44:20.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.877+0000 7f7caaa00640 1 --2- 172.21.15.28:0/2848102860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb41568e0 0x7f7cb40c2250 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c0097a0 tx=0x7f7c9c02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c93aa3b92dedfe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:20.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.877+0000 7f7cb2a00640 1 -- 172.21.15.28:0/2848102860 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c02ae50 con 0x7f7cb41568e0 2024-09-13T16:44:20.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.878+0000 7f7cb2a00640 1 -- 172.21.15.28:0/2848102860 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9c032440 con 0x7f7cb41568e0 2024-09-13T16:44:20.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.878+0000 7f7cb2a00640 1 -- 172.21.15.28:0/2848102860 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c0327b0 con 0x7f7cb41568e0 2024-09-13T16:44:20.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.878+0000 7f7cba600640 1 -- 172.21.15.28:0/2848102860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb41568e0 msgr2=0x7f7cb40c2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:20.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.878+0000 7f7cba600640 1 --2- 172.21.15.28:0/2848102860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb41568e0 0x7f7cb40c2250 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c0097a0 tx=0x7f7c9c02a100 comp rx=0 tx=0).stop 2024-09-13T16:44:20.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.878+0000 7f7cba600640 1 -- 172.21.15.28:0/2848102860 shutdown_connections 2024-09-13T16:44:20.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.878+0000 7f7cba600640 1 --2- 172.21.15.28:0/2848102860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7cb40c4d60 0x7f7cb40c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:20.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.878+0000 7f7cba600640 1 --2- 172.21.15.28:0/2848102860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb41568e0 0x7f7cb40c2250 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:20.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.878+0000 7f7cba600640 1 -- 172.21.15.28:0/2848102860 >> 172.21.15.28:0/2848102860 conn(0x7f7cb4066040 msgr2=0x7f7cb4069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:20.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.878+0000 7f7cba600640 1 -- 172.21.15.28:0/2848102860 shutdown_connections 2024-09-13T16:44:20.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.878+0000 7f7cba600640 1 -- 172.21.15.28:0/2848102860 wait complete. 2024-09-13T16:44:20.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.880+0000 7f7cba600640 1 Processor -- start 2024-09-13T16:44:20.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.880+0000 7f7cba600640 1 -- start start 2024-09-13T16:44:20.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.881+0000 7f7cba600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb40c4d60 0x7f7cb41597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:20.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.881+0000 7f7cba600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7cb4150ca0 0x7f7cb41efde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:20.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.881+0000 7f7cba600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7cb41568e0 0x7f7cb41f0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:20.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.881+0000 7f7cba600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb41ec220 con 0x7f7cb40c4d60 2024-09-13T16:44:20.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.881+0000 7f7cba600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb4147d40 con 0x7f7cb4150ca0 2024-09-13T16:44:20.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.881+0000 7f7cba600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb41590a0 con 0x7f7cb41568e0 2024-09-13T16:44:20.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.881+0000 7f7cb3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb40c4d60 0x7f7cb41597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:20.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.881+0000 7f7cb3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb40c4d60 0x7f7cb41597e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53588/0 (socket says 172.21.15.28:53588) 2024-09-13T16:44:20.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.881+0000 7f7cb3400640 1 -- 172.21.15.28:0/4097569852 learned_addr learned my addr 172.21.15.28:0/4097569852 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:20.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.881+0000 7f7caaa00640 1 --2- 172.21.15.28:0/4097569852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7cb4150ca0 0x7f7cb41efde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:20.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.881+0000 7f7cb3e00640 1 --2- 172.21.15.28:0/4097569852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7cb41568e0 0x7f7cb41f0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:20.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.881+0000 7f7cb3400640 1 -- 172.21.15.28:0/4097569852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7cb41568e0 msgr2=0x7f7cb41f0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:20.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.881+0000 7f7cb3400640 1 --2- 172.21.15.28:0/4097569852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7cb41568e0 0x7f7cb41f0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:20.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.881+0000 7f7cb3400640 1 -- 172.21.15.28:0/4097569852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7cb4150ca0 msgr2=0x7f7cb41efde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:20.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.881+0000 7f7cb3400640 1 --2- 172.21.15.28:0/4097569852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7cb4150ca0 0x7f7cb41efde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:20.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.881+0000 7f7cb3400640 1 -- 172.21.15.28:0/4097569852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c9c009400 con 0x7f7cb40c4d60 2024-09-13T16:44:20.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.882+0000 7f7cb3400640 1 --2- 172.21.15.28:0/4097569852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb40c4d60 0x7f7cb41597e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7ca400c430 tx=0x7f7ca400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:20.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.882+0000 7f7cabe00640 1 -- 172.21.15.28:0/4097569852 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4005ce0 con 0x7f7cb40c4d60 2024-09-13T16:44:20.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.882+0000 7f7cabe00640 1 -- 172.21.15.28:0/4097569852 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca4005e40 con 0x7f7cb40c4d60 2024-09-13T16:44:20.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.882+0000 7f7cabe00640 1 -- 172.21.15.28:0/4097569852 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40072f0 con 0x7f7cb40c4d60 2024-09-13T16:44:20.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.882+0000 7f7cba600640 1 -- 172.21.15.28:0/4097569852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb4148270 con 0x7f7cb40c4d60 2024-09-13T16:44:20.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.882+0000 7f7cba600640 1 -- 172.21.15.28:0/4097569852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb4159340 con 0x7f7cb40c4d60 2024-09-13T16:44:20.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.883+0000 7f7cba600640 1 -- 172.21.15.28:0/4097569852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb4159f00 con 0x7f7cb40c4d60 2024-09-13T16:44:20.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.884+0000 7f7cabe00640 1 -- 172.21.15.28:0/4097569852 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7ca4007450 con 0x7f7cb40c4d60 2024-09-13T16:44:20.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.885+0000 7f7cabe00640 1 --2- 172.21.15.28:0/4097569852 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7c9006ac00 0x7f7c9006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:20.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.885+0000 7f7cabe00640 1 -- 172.21.15.28:0/4097569852 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ca40889b0 con 0x7f7cb40c4d60 2024-09-13T16:44:20.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.885+0000 7f7caaa00640 1 --2- 172.21.15.28:0/4097569852 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7c9006ac00 0x7f7c9006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:20.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.886+0000 7f7caaa00640 1 --2- 172.21.15.28:0/4097569852 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7c9006ac00 0x7f7c9006d0c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c034000 tx=0x7f7c9c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:20.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:20.890+0000 7f7cabe00640 1 -- 172.21.15.28:0/4097569852 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7ca40576c0 con 0x7f7cb40c4d60 2024-09-13T16:44:21.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.069+0000 7f7cba600640 1 -- 172.21.15.28:0/4097569852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cb40c3110 con 0x7f7cb40c4d60 2024-09-13T16:44:21.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.070+0000 7f7cabe00640 1 -- 172.21.15.28:0/4097569852 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16481 (secure 0 0 0) 0x7f7ca4017020 con 0x7f7cb40c4d60 2024-09-13T16:44:21.071 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:21.072 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:43:55.971358+0000","last_up_change":"2024-09-13T16:43:49.476313+0000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:43:49.643606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:43:52.828510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:43:53.945874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6801","nonce":2964323435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6803","nonce":2964323435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6807","nonce":2964323435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6805","nonce":2964323435}]},"public_addr":"172.21.15.28:6801/2964323435","cluster_addr":"172.21.15.28:6803/2964323435","heartbeat_back_addr":"172.21.15.28:6807/2964323435","heartbeat_front_addr":"172.21.15.28:6805/2964323435","state":["exists","up"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6809","nonce":23768570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6811","nonce":23768570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6815","nonce":23768570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6813","nonce":23768570}]},"public_addr":"172.21.15.28:6809/23768570","cluster_addr":"172.21.15.28:6811/23768570","heartbeat_back_addr":"172.21.15.28:6815/23768570","heartbeat_front_addr":"172.21.15.28:6813/23768570","state":["exists","up"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6817","nonce":886887303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6819","nonce":886887303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6823","nonce":886887303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6821","nonce":886887303}]},"public_addr":"172.21.15.28:6817/886887303","cluster_addr":"172.21.15.28:6819/886887303","heartbeat_back_addr":"172.21.15.28:6823/886887303","heartbeat_front_addr":"172.21.15.28:6821/886887303","state":["exists","up"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6825","nonce":2616036674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6827","nonce":2616036674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6831","nonce":2616036674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6829","nonce":2616036674}]},"public_addr":"172.21.15.28:6825/2616036674","cluster_addr":"172.21.15.28:6827/2616036674","heartbeat_back_addr":"172.21.15.28:6831/2616036674","heartbeat_front_addr":"172.21.15.28:6829/2616036674","state":["exists","up"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6801","nonce":2817102450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6803","nonce":2817102450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6807","nonce":2817102450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6805","nonce":2817102450}]},"public_addr":"172.21.15.191:6801/2817102450","cluster_addr":"172.21.15.191:6803/2817102450","heartbeat_back_addr":"172.21.15.191:6807/2817102450","heartbeat_front_addr":"172.21.15.191:6805/2817102450","state":["exists","up"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6817","nonce":2503027971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6819","nonce":2503027971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6823","nonce":2503027971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6821","nonce":2503027971}]},"public_addr":"172.21.15.191:6817/2503027971","cluster_addr":"172.21.15.191:6819/2503027971","heartbeat_back_addr":"172.21.15.191:6823/2503027971","heartbeat_front_addr":"172.21.15.191:6821/2503027971","state":["exists","up"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6809","nonce":3006595088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6811","nonce":3006595088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6815","nonce":3006595088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6813","nonce":3006595088}]},"public_addr":"172.21.15.191:6809/3006595088","cluster_addr":"172.21.15.191:6811/3006595088","heartbeat_back_addr":"172.21.15.191:6815/3006595088","heartbeat_front_addr":"172.21.15.191:6813/3006595088","state":["exists","up"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6825","nonce":1477327720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6827","nonce":1477327720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6831","nonce":1477327720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6829","nonce":1477327720}]},"public_addr":"172.21.15.191:6825/1477327720","cluster_addr":"172.21.15.191:6827/1477327720","heartbeat_back_addr":"172.21.15.191:6831/1477327720","heartbeat_front_addr":"172.21.15.191:6829/1477327720","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.917250+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.954351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.017329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.046221+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.707467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.726947+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.782321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.764850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:44:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.072+0000 7f7cba600640 1 -- 172.21.15.28:0/4097569852 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7c9006ac00 msgr2=0x7f7c9006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.072+0000 7f7cba600640 1 --2- 172.21.15.28:0/4097569852 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7c9006ac00 0x7f7c9006d0c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c034000 tx=0x7f7c9c02e000 comp rx=0 tx=0).stop 2024-09-13T16:44:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.072+0000 7f7cba600640 1 -- 172.21.15.28:0/4097569852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb40c4d60 msgr2=0x7f7cb41597e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.072+0000 7f7cba600640 1 --2- 172.21.15.28:0/4097569852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb40c4d60 0x7f7cb41597e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7ca400c430 tx=0x7f7ca400c8f0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.072+0000 7f7cba600640 1 -- 172.21.15.28:0/4097569852 shutdown_connections 2024-09-13T16:44:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.072+0000 7f7cba600640 1 --2- 172.21.15.28:0/4097569852 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7c9006ac00 0x7f7c9006d0c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.072+0000 7f7cba600640 1 --2- 172.21.15.28:0/4097569852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7cb41568e0 0x7f7cb41f0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.072+0000 7f7cba600640 1 --2- 172.21.15.28:0/4097569852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7cb4150ca0 0x7f7cb41efde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.072+0000 7f7cba600640 1 --2- 172.21.15.28:0/4097569852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb40c4d60 0x7f7cb41597e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.072+0000 7f7cba600640 1 -- 172.21.15.28:0/4097569852 >> 172.21.15.28:0/4097569852 conn(0x7f7cb4066040 msgr2=0x7f7cb40c63f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.072+0000 7f7cba600640 1 -- 172.21.15.28:0/4097569852 shutdown_connections 2024-09-13T16:44:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.072+0000 7f7cba600640 1 -- 172.21.15.28:0/4097569852 wait complete. 2024-09-13T16:44:21.090 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:43:49.643606+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:43:52.828510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:43:53.945874+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:44:21.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:44:21.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.246+0000 7fb33f200640 1 Processor -- start 2024-09-13T16:44:21.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.247+0000 7fb33f200640 1 -- start start 2024-09-13T16:44:21.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.248+0000 7fb33f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb340152b80 0x7fb34015be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:21.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.248+0000 7fb33f200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb3400c9aa0 0x7fb340159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:21.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.248+0000 7fb33f200640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3400c3710 con 0x7fb3400aedf0 2024-09-13T16:44:21.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.248+0000 7fb33f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb340158f00 con 0x7fb340152b80 2024-09-13T16:44:21.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.248+0000 7fb33f200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3400c8ee0 con 0x7fb3400c9aa0 2024-09-13T16:44:21.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.248+0000 7fb33de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb340152b80 0x7fb34015be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:21.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.248+0000 7fb33de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb340152b80 0x7fb34015be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53594/0 (socket says 172.21.15.28:53594) 2024-09-13T16:44:21.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.248+0000 7fb33de00640 1 -- 172.21.15.28:0/46208050 learned_addr learned my addr 172.21.15.28:0/46208050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:21.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.248+0000 7fb337e00640 1 --2- 172.21.15.28:0/46208050 >> v2:172.21.15.191:3300/0 conn(0x7fb3400c9aa0 0x7fb340159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:21.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.248+0000 7fb33de00640 1 -- 172.21.15.28:0/46208050 >> v2:172.21.15.191:3300/0 conn(0x7fb3400c9aa0 msgr2=0x7fb340159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:21.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.248+0000 7fb33de00640 1 --2- 172.21.15.28:0/46208050 >> v2:172.21.15.191:3300/0 conn(0x7fb3400c9aa0 0x7fb340159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.248+0000 7fb33de00640 1 -- 172.21.15.28:0/46208050 >> v1:172.21.15.191:6789/0 conn(0x7fb3400aedf0 legacy=0x7fb34006d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:21.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.249+0000 7fb33de00640 1 -- 172.21.15.28:0/46208050 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3400af260 con 0x7fb340152b80 2024-09-13T16:44:21.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.249+0000 7fb33de00640 1 --2- 172.21.15.28:0/46208050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb340152b80 0x7fb34015be40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb3400c94d0 tx=0x7fb33002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cf2d59fa1840631 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:21.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.250+0000 7fb33d400640 1 -- 172.21.15.28:0/46208050 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb330032440 con 0x7fb340152b80 2024-09-13T16:44:21.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.250+0000 7fb33d400640 1 -- 172.21.15.28:0/46208050 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3300325a0 con 0x7fb340152b80 2024-09-13T16:44:21.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.250+0000 7fb33d400640 1 -- 172.21.15.28:0/46208050 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb330032930 con 0x7fb340152b80 2024-09-13T16:44:21.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.250+0000 7fb33f200640 1 -- 172.21.15.28:0/46208050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb340152b80 msgr2=0x7fb34015be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:21.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.250+0000 7fb33f200640 1 --2- 172.21.15.28:0/46208050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb340152b80 0x7fb34015be40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb3400c94d0 tx=0x7fb33002a270 comp rx=0 tx=0).stop 2024-09-13T16:44:21.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.250+0000 7fb33f200640 1 -- 172.21.15.28:0/46208050 shutdown_connections 2024-09-13T16:44:21.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.250+0000 7fb33f200640 1 --2- 172.21.15.28:0/46208050 >> v2:172.21.15.191:3300/0 conn(0x7fb3400c9aa0 0x7fb340159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.250+0000 7fb33f200640 1 --2- 172.21.15.28:0/46208050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb340152b80 0x7fb34015be40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.250+0000 7fb33f200640 1 -- 172.21.15.28:0/46208050 >> 172.21.15.28:0/46208050 conn(0x7fb340065c50 msgr2=0x7fb340066060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:21.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.251+0000 7fb33f200640 1 -- 172.21.15.28:0/46208050 shutdown_connections 2024-09-13T16:44:21.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.251+0000 7fb33f200640 1 -- 172.21.15.28:0/46208050 wait complete. 2024-09-13T16:44:21.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.252+0000 7fb33f200640 1 Processor -- start 2024-09-13T16:44:21.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.252+0000 7fb33f200640 1 -- start start 2024-09-13T16:44:21.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.253+0000 7fb33f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3400aedf0 0x7fb3401548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:21.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.253+0000 7fb33f200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3400c9aa0 0x7fb34015ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:21.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.253+0000 7fb33f200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb340152b80 0x7fb34014bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.253+0000 7fb33f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34015ada0 con 0x7fb3400aedf0 2024-09-13T16:44:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.253+0000 7fb33f200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb340155170 con 0x7fb340152b80 2024-09-13T16:44:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.253+0000 7fb33f200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34015b600 con 0x7fb3400c9aa0 2024-09-13T16:44:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.253+0000 7fb33de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3400aedf0 0x7fb3401548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.253+0000 7fb33de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3400aedf0 0x7fb3401548d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53600/0 (socket says 172.21.15.28:53600) 2024-09-13T16:44:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.253+0000 7fb33de00640 1 -- 172.21.15.28:0/2044836047 learned_addr learned my addr 172.21.15.28:0/2044836047 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.253+0000 7fb337e00640 1 --2- 172.21.15.28:0/2044836047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3400c9aa0 0x7fb34015ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.254+0000 7fb33e800640 1 --2- 172.21.15.28:0/2044836047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb340152b80 0x7fb34014bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.254+0000 7fb33de00640 1 -- 172.21.15.28:0/2044836047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3400c9aa0 msgr2=0x7fb34015ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.254+0000 7fb33de00640 1 --2- 172.21.15.28:0/2044836047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3400c9aa0 0x7fb34015ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.254+0000 7fb33de00640 1 -- 172.21.15.28:0/2044836047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb340152b80 msgr2=0x7fb34014bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.254+0000 7fb33de00640 1 --2- 172.21.15.28:0/2044836047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb340152b80 0x7fb34014bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.254+0000 7fb33de00640 1 -- 172.21.15.28:0/2044836047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3300094d0 con 0x7fb3400aedf0 2024-09-13T16:44:21.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.254+0000 7fb33de00640 1 --2- 172.21.15.28:0/2044836047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3400aedf0 0x7fb3401548d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb33002a2b0 tx=0x7fb3300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.254+0000 7fb336000640 1 -- 172.21.15.28:0/2044836047 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb330032440 con 0x7fb3400aedf0 2024-09-13T16:44:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.254+0000 7fb336000640 1 -- 172.21.15.28:0/2044836047 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb330030980 con 0x7fb3400aedf0 2024-09-13T16:44:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.254+0000 7fb33f200640 1 -- 172.21.15.28:0/2044836047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb340159d90 con 0x7fb3400aedf0 2024-09-13T16:44:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.254+0000 7fb336000640 1 -- 172.21.15.28:0/2044836047 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb330030d10 con 0x7fb3400aedf0 2024-09-13T16:44:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.255+0000 7fb33f200640 1 -- 172.21.15.28:0/2044836047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34015afb0 con 0x7fb3400aedf0 2024-09-13T16:44:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.256+0000 7fb33f200640 1 -- 172.21.15.28:0/2044836047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb340067460 con 0x7fb3400aedf0 2024-09-13T16:44:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.256+0000 7fb336000640 1 -- 172.21.15.28:0/2044836047 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb330039040 con 0x7fb3400aedf0 2024-09-13T16:44:21.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.257+0000 7fb336000640 1 --2- 172.21.15.28:0/2044836047 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb31c06ac00 0x7fb31c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:21.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.257+0000 7fb336000640 1 -- 172.21.15.28:0/2044836047 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3300a9b70 con 0x7fb3400aedf0 2024-09-13T16:44:21.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.257+0000 7fb337e00640 1 --2- 172.21.15.28:0/2044836047 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb31c06ac00 0x7fb31c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:21.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.258+0000 7fb337e00640 1 --2- 172.21.15.28:0/2044836047 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb31c06ac00 0x7fb31c06d0c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb34014ecd0 tx=0x7fb328009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:21.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.261+0000 7fb336000640 1 -- 172.21.15.28:0/2044836047 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb330074df0 con 0x7fb3400aedf0 2024-09-13T16:44:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.439+0000 7fb33f200640 1 -- 172.21.15.28:0/2044836047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb340158b90 con 0x7fb3400aedf0 2024-09-13T16:44:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.440+0000 7fb336000640 1 -- 172.21.15.28:0/2044836047 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb3300410a0 con 0x7fb3400aedf0 2024-09-13T16:44:21.441 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T16:44:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.442+0000 7fb33f200640 1 -- 172.21.15.28:0/2044836047 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb31c06ac00 msgr2=0x7fb31c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.442+0000 7fb33f200640 1 --2- 172.21.15.28:0/2044836047 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb31c06ac00 0x7fb31c06d0c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb34014ecd0 tx=0x7fb328009190 comp rx=0 tx=0).stop 2024-09-13T16:44:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.442+0000 7fb33f200640 1 -- 172.21.15.28:0/2044836047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3400aedf0 msgr2=0x7fb3401548d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.442+0000 7fb33f200640 1 --2- 172.21.15.28:0/2044836047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3400aedf0 0x7fb3401548d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb33002a2b0 tx=0x7fb3300046c0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.442+0000 7fb33f200640 1 -- 172.21.15.28:0/2044836047 shutdown_connections 2024-09-13T16:44:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.442+0000 7fb33f200640 1 --2- 172.21.15.28:0/2044836047 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb31c06ac00 0x7fb31c06d0c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.442+0000 7fb33f200640 1 --2- 172.21.15.28:0/2044836047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb340152b80 0x7fb34014bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.442+0000 7fb33f200640 1 --2- 172.21.15.28:0/2044836047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3400c9aa0 0x7fb34015ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.442+0000 7fb33f200640 1 --2- 172.21.15.28:0/2044836047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3400aedf0 0x7fb3401548d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.442+0000 7fb33f200640 1 -- 172.21.15.28:0/2044836047 >> 172.21.15.28:0/2044836047 conn(0x7fb340065c50 msgr2=0x7fb34006b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.442+0000 7fb33f200640 1 -- 172.21.15.28:0/2044836047 shutdown_connections 2024-09-13T16:44:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.442+0000 7fb33f200640 1 -- 172.21.15.28:0/2044836047 wait complete. 2024-09-13T16:44:21.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:44:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.618+0000 7f8fbfe00640 1 Processor -- start 2024-09-13T16:44:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.618+0000 7f8fbfe00640 1 -- start start 2024-09-13T16:44:21.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.619+0000 7f8fbfe00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8fb8152d50 0x7f8fb80c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:21.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.619+0000 7f8fbfe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fb80c6cf0 con 0x7f8fb80c5340 2024-09-13T16:44:21.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.619+0000 7f8fbfe00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fb815cbc0 con 0x7f8fb80c5770 2024-09-13T16:44:21.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.619+0000 7f8fbfe00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fb815c080 con 0x7f8fb8152d50 2024-09-13T16:44:21.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.619+0000 7f8fbca00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8fb80c5340 0x7f8fb806d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60974/0 (socket says 172.21.15.28:60974) 2024-09-13T16:44:21.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.619+0000 7f8fbca00640 1 -- 172.21.15.28:0/3013931649 learned_addr learned my addr 172.21.15.28:0/3013931649 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:21.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.619+0000 7f8fbd400640 1 --2- 172.21.15.28:0/3013931649 >> v2:172.21.15.191:3300/0 conn(0x7f8fb8152d50 0x7f8fb80c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:21.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.620+0000 7f8fb7e00640 1 -- 172.21.15.28:0/3013931649 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752174998 0 0) 0x7f8fa4001110 con 0x7f8fb80c5340 2024-09-13T16:44:21.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.620+0000 7f8fb7e00640 1 -- 172.21.15.28:0/3013931649 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa0003520 con 0x7f8fb80c5340 2024-09-13T16:44:21.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.620+0000 7f8fb7e00640 1 -- 172.21.15.28:0/3013931649 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330323848 0 0) 0x7f8fb815cbc0 con 0x7f8fb80c5770 2024-09-13T16:44:21.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.620+0000 7f8fbd400640 1 -- 172.21.15.28:0/3013931649 >> v1:172.21.15.191:6789/0 conn(0x7f8fb80c5770 legacy=0x7f8fb806fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:21.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.620+0000 7f8fbd400640 1 -- 172.21.15.28:0/3013931649 >> v1:172.21.15.28:6789/0 conn(0x7f8fb80c5340 legacy=0x7f8fb806d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:21.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.620+0000 7f8fbd400640 1 -- 172.21.15.28:0/3013931649 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb80c5d10 con 0x7f8fb8152d50 2024-09-13T16:44:21.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.621+0000 7f8fbd400640 1 --2- 172.21.15.28:0/3013931649 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fb8152d50 0x7f8fb80c67b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8fac0097a0 tx=0x7f8fac02a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4618333d25385521 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:21.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.621+0000 7f8fb7e00640 1 -- 172.21.15.28:0/3013931649 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fac032640 con 0x7f8fb8152d50 2024-09-13T16:44:21.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.621+0000 7f8fb7e00640 1 -- 172.21.15.28:0/3013931649 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fac0327a0 con 0x7f8fb8152d50 2024-09-13T16:44:21.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.621+0000 7f8fb7e00640 1 -- 172.21.15.28:0/3013931649 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fac032b10 con 0x7f8fb8152d50 2024-09-13T16:44:21.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.621+0000 7f8fbfe00640 1 -- 172.21.15.28:0/3013931649 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fb8152d50 msgr2=0x7f8fb80c67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:21.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.621+0000 7f8fbfe00640 1 --2- 172.21.15.28:0/3013931649 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fb8152d50 0x7f8fb80c67b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8fac0097a0 tx=0x7f8fac02a2c0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.622+0000 7f8fbfe00640 1 -- 172.21.15.28:0/3013931649 shutdown_connections 2024-09-13T16:44:21.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.622+0000 7f8fbfe00640 1 --2- 172.21.15.28:0/3013931649 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fb8152d50 0x7f8fb80c67b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.622+0000 7f8fbfe00640 1 -- 172.21.15.28:0/3013931649 >> 172.21.15.28:0/3013931649 conn(0x7f8fb80ca6b0 msgr2=0x7f8fb80caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:21.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.622+0000 7f8fbfe00640 1 -- 172.21.15.28:0/3013931649 shutdown_connections 2024-09-13T16:44:21.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.622+0000 7f8fbfe00640 1 -- 172.21.15.28:0/3013931649 wait complete. 2024-09-13T16:44:21.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.623+0000 7f8fbfe00640 1 Processor -- start 2024-09-13T16:44:21.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.623+0000 7f8fbfe00640 1 -- start start 2024-09-13T16:44:21.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.624+0000 7f8fbfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fb80c5340 0x7f8fb81ef730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:21.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.624+0000 7f8fbfe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fb80c5770 0x7f8fb81e1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.624+0000 7f8fbfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb8152d50 0x7f8fb81e2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.624+0000 7f8fbfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb80c3630 con 0x7f8fb8152d50 2024-09-13T16:44:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.624+0000 7f8fbfe00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb80c3890 con 0x7f8fb80c5340 2024-09-13T16:44:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.624+0000 7f8fbfe00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb81e2b20 con 0x7f8fb80c5770 2024-09-13T16:44:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.624+0000 7f8fbd400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fb80c5340 0x7f8fb81ef730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.624+0000 7f8fbde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb8152d50 0x7f8fb81e2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.625+0000 7f8fbd400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fb80c5340 0x7f8fb81ef730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:50986/0 (socket says 172.21.15.28:50986) 2024-09-13T16:44:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.625+0000 7f8fbd400640 1 -- 172.21.15.28:0/3999812584 learned_addr learned my addr 172.21.15.28:0/3999812584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.625+0000 7f8fbca00640 1 --2- 172.21.15.28:0/3999812584 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fb80c5770 0x7f8fb81e1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.625+0000 7f8fbde00640 1 -- 172.21.15.28:0/3999812584 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fb80c5770 msgr2=0x7f8fb81e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.625+0000 7f8fbde00640 1 --2- 172.21.15.28:0/3999812584 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fb80c5770 0x7f8fb81e1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.625+0000 7f8fbde00640 1 -- 172.21.15.28:0/3999812584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fb80c5340 msgr2=0x7f8fb81ef730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:21.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.625+0000 7f8fbde00640 1 --2- 172.21.15.28:0/3999812584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fb80c5340 0x7f8fb81ef730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.625+0000 7f8fbde00640 1 -- 172.21.15.28:0/3999812584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fac009400 con 0x7f8fb8152d50 2024-09-13T16:44:21.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.625+0000 7f8fbd400640 1 --2- 172.21.15.28:0/3999812584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fb80c5340 0x7f8fb81ef730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:21.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.625+0000 7f8fbde00640 1 --2- 172.21.15.28:0/3999812584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb8152d50 0x7f8fb81e2480 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8fb000c6a0 tx=0x7f8fb000cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.625+0000 7f8fb5600640 1 -- 172.21.15.28:0/3999812584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb0003b70 con 0x7f8fb8152d50 2024-09-13T16:44:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.626+0000 7f8fb5600640 1 -- 172.21.15.28:0/3999812584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb0003cf0 con 0x7f8fb8152d50 2024-09-13T16:44:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.626+0000 7f8fbfe00640 1 -- 172.21.15.28:0/3999812584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fb81e2dc0 con 0x7f8fb8152d50 2024-09-13T16:44:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.626+0000 7f8fb5600640 1 -- 172.21.15.28:0/3999812584 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb00052b0 con 0x7f8fb8152d50 2024-09-13T16:44:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.626+0000 7f8fbfe00640 1 -- 172.21.15.28:0/3999812584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fb80cae60 con 0x7f8fb8152d50 2024-09-13T16:44:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.627+0000 7f8fbfe00640 1 -- 172.21.15.28:0/3999812584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fb806df20 con 0x7f8fb8152d50 2024-09-13T16:44:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.627+0000 7f8fb5600640 1 -- 172.21.15.28:0/3999812584 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8fb0005410 con 0x7f8fb8152d50 2024-09-13T16:44:21.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.628+0000 7f8fb5600640 1 --2- 172.21.15.28:0/3999812584 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8f9806ac00 0x7f8f9806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:21.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.628+0000 7f8fb5600640 1 -- 172.21.15.28:0/3999812584 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8fb00891a0 con 0x7f8fb8152d50 2024-09-13T16:44:21.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.629+0000 7f8fbd400640 1 --2- 172.21.15.28:0/3999812584 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8f9806ac00 0x7f8f9806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.629+0000 7f8fbd400640 1 --2- 172.21.15.28:0/3999812584 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8f9806ac00 0x7f8f9806d0c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8fac0097a0 tx=0x7f8fac02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:21.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.632+0000 7f8fb5600640 1 -- 172.21.15.28:0/3999812584 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8fb0054280 con 0x7f8fb8152d50 2024-09-13T16:44:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.812+0000 7f8fbfe00640 1 -- 172.21.15.28:0/3999812584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8fb81e3b60 con 0x7f8fb8152d50 2024-09-13T16:44:21.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.812+0000 7f8fb5600640 1 -- 172.21.15.28:0/3999812584 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8fb0003e60 con 0x7f8fb8152d50 2024-09-13T16:44:21.814 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T16:44:21.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.814+0000 7f8fbfe00640 1 -- 172.21.15.28:0/3999812584 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8f9806ac00 msgr2=0x7f8f9806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:21.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.814+0000 7f8fbfe00640 1 --2- 172.21.15.28:0/3999812584 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8f9806ac00 0x7f8f9806d0c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8fac0097a0 tx=0x7f8fac02e000 comp rx=0 tx=0).stop 2024-09-13T16:44:21.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.814+0000 7f8fbfe00640 1 -- 172.21.15.28:0/3999812584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb8152d50 msgr2=0x7f8fb81e2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:21.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.814+0000 7f8fbfe00640 1 --2- 172.21.15.28:0/3999812584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb8152d50 0x7f8fb81e2480 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8fb000c6a0 tx=0x7f8fb000cb60 comp rx=0 tx=0).stop 2024-09-13T16:44:21.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.814+0000 7f8fbfe00640 1 -- 172.21.15.28:0/3999812584 shutdown_connections 2024-09-13T16:44:21.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.814+0000 7f8fbfe00640 1 --2- 172.21.15.28:0/3999812584 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8f9806ac00 0x7f8f9806d0c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.814+0000 7f8fbfe00640 1 --2- 172.21.15.28:0/3999812584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb8152d50 0x7f8fb81e2480 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.814+0000 7f8fbfe00640 1 --2- 172.21.15.28:0/3999812584 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8fb80c5770 0x7f8fb81e1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.814+0000 7f8fbfe00640 1 --2- 172.21.15.28:0/3999812584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8fb80c5340 0x7f8fb81ef730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.814+0000 7f8fbfe00640 1 -- 172.21.15.28:0/3999812584 >> 172.21.15.28:0/3999812584 conn(0x7f8fb80ca6b0 msgr2=0x7f8fb814f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:21.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.815+0000 7f8fbfe00640 1 -- 172.21.15.28:0/3999812584 shutdown_connections 2024-09-13T16:44:21.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.815+0000 7f8fbfe00640 1 -- 172.21.15.28:0/3999812584 wait complete. 2024-09-13T16:44:21.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:44:21.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.991+0000 7f3dba000640 1 Processor -- start 2024-09-13T16:44:21.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.991+0000 7f3dba000640 1 -- start start 2024-09-13T16:44:21.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.992+0000 7f3dba000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3db40c7f10 0x7f3db40c6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:21.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.992+0000 7f3dba000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3db40c5c00 0x7f3db4158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:21.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.992+0000 7f3dba000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3db40c82f0 0x7f3db41590f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:21.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.992+0000 7f3dba000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3db4159630 con 0x7f3db40c5c00 2024-09-13T16:44:21.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.992+0000 7f3dba000640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3db406f760 con 0x7f3db40c82f0 2024-09-13T16:44:21.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.992+0000 7f3dba000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db40c6e70 con 0x7f3db40c7f10 2024-09-13T16:44:21.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.992+0000 7f3db3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3db40c5c00 0x7f3db4158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:21.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.992+0000 7f3db3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3db40c5c00 0x7f3db4158bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53614/0 (socket says 172.21.15.28:53614) 2024-09-13T16:44:21.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.992+0000 7f3db3e00640 1 -- 172.21.15.28:0/3779248582 learned_addr learned my addr 172.21.15.28:0/3779248582 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:21.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.993+0000 7f3db8c00640 1 --2- 172.21.15.28:0/3779248582 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3db40c7f10 0x7f3db40c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:21.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.993+0000 7f3db9600640 1 --2- 172.21.15.28:0/3779248582 >> v2:172.21.15.191:3300/0 conn(0x7f3db40c82f0 0x7f3db41590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:21.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.993+0000 7f3db3e00640 1 -- 172.21.15.28:0/3779248582 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3db40c7f10 msgr2=0x7f3db40c6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:21.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.993+0000 7f3db3e00640 1 --2- 172.21.15.28:0/3779248582 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3db40c7f10 0x7f3db40c6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.993+0000 7f3db3e00640 1 -- 172.21.15.28:0/3779248582 >> v2:172.21.15.191:3300/0 conn(0x7f3db40c82f0 msgr2=0x7f3db41590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:21.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.993+0000 7f3db3e00640 1 --2- 172.21.15.28:0/3779248582 >> v2:172.21.15.191:3300/0 conn(0x7f3db40c82f0 0x7f3db41590f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.993+0000 7f3db3e00640 1 -- 172.21.15.28:0/3779248582 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db40c7110 con 0x7f3db40c5c00 2024-09-13T16:44:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.994+0000 7f3db3e00640 1 --2- 172.21.15.28:0/3779248582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3db40c5c00 0x7f3db4158bb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3db40c31e0 tx=0x7f3da002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c99972c3cffd5e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.994+0000 7f3db3400640 1 -- 172.21.15.28:0/3779248582 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da0032440 con 0x7f3db40c5c00 2024-09-13T16:44:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.994+0000 7f3db3400640 1 -- 172.21.15.28:0/3779248582 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da00325a0 con 0x7f3db40c5c00 2024-09-13T16:44:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.994+0000 7f3db3400640 1 -- 172.21.15.28:0/3779248582 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da0032930 con 0x7f3db40c5c00 2024-09-13T16:44:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.995+0000 7f3dba000640 1 -- 172.21.15.28:0/3779248582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3db40c5c00 msgr2=0x7f3db4158bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.995+0000 7f3dba000640 1 --2- 172.21.15.28:0/3779248582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3db40c5c00 0x7f3db4158bb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3db40c31e0 tx=0x7f3da002a270 comp rx=0 tx=0).stop 2024-09-13T16:44:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.995+0000 7f3dba000640 1 -- 172.21.15.28:0/3779248582 shutdown_connections 2024-09-13T16:44:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.995+0000 7f3dba000640 1 --2- 172.21.15.28:0/3779248582 >> v2:172.21.15.191:3300/0 conn(0x7f3db40c82f0 0x7f3db41590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.995+0000 7f3dba000640 1 --2- 172.21.15.28:0/3779248582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3db40c5c00 0x7f3db4158bb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.995+0000 7f3dba000640 1 --2- 172.21.15.28:0/3779248582 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3db40c7f10 0x7f3db40c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.995+0000 7f3dba000640 1 -- 172.21.15.28:0/3779248582 >> 172.21.15.28:0/3779248582 conn(0x7f3db406af90 msgr2=0x7f3db406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.995+0000 7f3dba000640 1 -- 172.21.15.28:0/3779248582 shutdown_connections 2024-09-13T16:44:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.995+0000 7f3dba000640 1 -- 172.21.15.28:0/3779248582 wait complete. 2024-09-13T16:44:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.996+0000 7f3dba000640 1 Processor -- start 2024-09-13T16:44:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.997+0000 7f3dba000640 1 -- start start 2024-09-13T16:44:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.997+0000 7f3dba000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3db40c5c00 0x7f3db4124480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.997+0000 7f3dba000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3db40c7f10 0x7f3db41f1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.997+0000 7f3dba000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3db40c82f0 0x7f3db40c0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.997+0000 7f3dba000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db40be5e0 con 0x7f3db40c82f0 2024-09-13T16:44:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.997+0000 7f3dba000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db40be0d0 con 0x7f3db40c7f10 2024-09-13T16:44:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.997+0000 7f3dba000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db40c0930 con 0x7f3db40c5c00 2024-09-13T16:44:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.998+0000 7f3db9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3db40c82f0 0x7f3db40c0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.998+0000 7f3db8c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3db40c5c00 0x7f3db4124480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.998+0000 7f3db3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3db40c7f10 0x7f3db41f1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.998+0000 7f3db9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3db40c82f0 0x7f3db40c0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53618/0 (socket says 172.21.15.28:53618) 2024-09-13T16:44:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.998+0000 7f3db8c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3db40c5c00 0x7f3db4124480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:60432/0 (socket says 172.21.15.28:60432) 2024-09-13T16:44:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.998+0000 7f3db9600640 1 -- 172.21.15.28:0/933749872 learned_addr learned my addr 172.21.15.28:0/933749872 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.998+0000 7f3db9600640 1 -- 172.21.15.28:0/933749872 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3db40c5c00 msgr2=0x7f3db4124480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.998+0000 7f3db9600640 1 --2- 172.21.15.28:0/933749872 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3db40c5c00 0x7f3db4124480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.998+0000 7f3db9600640 1 -- 172.21.15.28:0/933749872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3db40c7f10 msgr2=0x7f3db41f1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.998+0000 7f3db9600640 1 --2- 172.21.15.28:0/933749872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3db40c7f10 0x7f3db41f1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.998+0000 7f3db9600640 1 -- 172.21.15.28:0/933749872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da00094d0 con 0x7f3db40c82f0 2024-09-13T16:44:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.998+0000 7f3db9600640 1 --2- 172.21.15.28:0/933749872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3db40c82f0 0x7f3db40c0dc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009d30 tx=0x7f3dac00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.999+0000 7f3db0c00640 1 -- 172.21.15.28:0/933749872 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac009ba0 con 0x7f3db40c82f0 2024-09-13T16:44:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.999+0000 7f3db0c00640 1 -- 172.21.15.28:0/933749872 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dac00f040 con 0x7f3db40c82f0 2024-09-13T16:44:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.999+0000 7f3db0c00640 1 -- 172.21.15.28:0/933749872 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac0146c0 con 0x7f3db40c82f0 2024-09-13T16:44:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.999+0000 7f3dba000640 1 -- 172.21.15.28:0/933749872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db40be300 con 0x7f3db40c82f0 2024-09-13T16:44:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:21.999+0000 7f3dba000640 1 -- 172.21.15.28:0/933749872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db41f24c0 con 0x7f3db40c82f0 2024-09-13T16:44:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.000+0000 7f3dba000640 1 -- 172.21.15.28:0/933749872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db40c0ae0 con 0x7f3db40c82f0 2024-09-13T16:44:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.001+0000 7f3db0c00640 1 -- 172.21.15.28:0/933749872 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3dac014820 con 0x7f3db40c82f0 2024-09-13T16:44:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.001+0000 7f3db0c00640 1 --2- 172.21.15.28:0/933749872 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3d9406ac30 0x7f3d9406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.002+0000 7f3db0c00640 1 -- 172.21.15.28:0/933749872 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3dac089fd0 con 0x7f3db40c82f0 2024-09-13T16:44:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.002+0000 7f3db8c00640 1 --2- 172.21.15.28:0/933749872 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3d9406ac30 0x7f3d9406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:22.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.002+0000 7f3db8c00640 1 --2- 172.21.15.28:0/933749872 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3d9406ac30 0x7f3d9406d0f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009ac0 tx=0x7f3da8009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.006+0000 7f3db0c00640 1 -- 172.21.15.28:0/933749872 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3dac055250 con 0x7f3db40c82f0 2024-09-13T16:44:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.184+0000 7f3dba000640 1 -- 172.21.15.28:0/933749872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3db415cca0 con 0x7f3db40c82f0 2024-09-13T16:44:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.185+0000 7f3db0c00640 1 -- 172.21.15.28:0/933749872 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3dac018020 con 0x7f3db40c82f0 2024-09-13T16:44:22.186 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T16:44:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.187+0000 7f3dba000640 1 -- 172.21.15.28:0/933749872 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3d9406ac30 msgr2=0x7f3d9406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.187+0000 7f3dba000640 1 --2- 172.21.15.28:0/933749872 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3d9406ac30 0x7f3d9406d0f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009ac0 tx=0x7f3da8009190 comp rx=0 tx=0).stop 2024-09-13T16:44:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.187+0000 7f3dba000640 1 -- 172.21.15.28:0/933749872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3db40c82f0 msgr2=0x7f3db40c0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.187+0000 7f3dba000640 1 --2- 172.21.15.28:0/933749872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3db40c82f0 0x7f3db40c0dc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009d30 tx=0x7f3dac00c490 comp rx=0 tx=0).stop 2024-09-13T16:44:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.187+0000 7f3dba000640 1 -- 172.21.15.28:0/933749872 shutdown_connections 2024-09-13T16:44:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.187+0000 7f3dba000640 1 --2- 172.21.15.28:0/933749872 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3d9406ac30 0x7f3d9406d0f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.187+0000 7f3dba000640 1 --2- 172.21.15.28:0/933749872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3db40c82f0 0x7f3db40c0dc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.187+0000 7f3dba000640 1 --2- 172.21.15.28:0/933749872 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3db40c7f10 0x7f3db41f1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.187+0000 7f3dba000640 1 --2- 172.21.15.28:0/933749872 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3db40c5c00 0x7f3db4124480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.187+0000 7f3dba000640 1 -- 172.21.15.28:0/933749872 >> 172.21.15.28:0/933749872 conn(0x7f3db406af90 msgr2=0x7f3db4071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.187+0000 7f3dba000640 1 -- 172.21.15.28:0/933749872 shutdown_connections 2024-09-13T16:44:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.187+0000 7f3dba000640 1 -- 172.21.15.28:0/933749872 wait complete. 2024-09-13T16:44:22.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:44:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.357+0000 7f1201e00640 1 Processor -- start 2024-09-13T16:44:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.357+0000 7f1201e00640 1 -- start start 2024-09-13T16:44:22.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.359+0000 7f1201e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f11fc0cb4c0 0x7f11fc155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:22.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.359+0000 7f1201e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11fc0cb8a0 0x7f11fc156220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:22.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.359+0000 7f1201e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f11fc06df20 0x7f11fc158c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:22.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.359+0000 7f1201e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11fc06f520 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.359+0000 7f1201e00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11fc06ec30 con 0x7f11fc06df20 2024-09-13T16:44:22.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.359+0000 7f1201e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc06aa50 con 0x7f11fc0cb8a0 2024-09-13T16:44:22.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.359+0000 7f1200a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f11fc0cb4c0 0x7f11fc155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:22.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.359+0000 7f1200a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f11fc0cb4c0 0x7f11fc155ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53630/0 (socket says 172.21.15.28:53630) 2024-09-13T16:44:22.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.359+0000 7f1200a00640 1 -- 172.21.15.28:0/2927039198 learned_addr learned my addr 172.21.15.28:0/2927039198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:22.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.359+0000 7f1201400640 1 --2- 172.21.15.28:0/2927039198 >> v2:172.21.15.191:3300/0 conn(0x7f11fc06df20 0x7f11fc158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:22.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.359+0000 7f11f3e00640 1 --2- 172.21.15.28:0/2927039198 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11fc0cb8a0 0x7f11fc156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.359+0000 7f1200a00640 1 -- 172.21.15.28:0/2927039198 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11fc0cb8a0 msgr2=0x7f11fc156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.359+0000 7f1200a00640 1 --2- 172.21.15.28:0/2927039198 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11fc0cb8a0 0x7f11fc156220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.359+0000 7f1200a00640 1 -- 172.21.15.28:0/2927039198 >> v2:172.21.15.191:3300/0 conn(0x7f11fc06df20 msgr2=0x7f11fc158c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.359+0000 7f1200a00640 1 --2- 172.21.15.28:0/2927039198 >> v2:172.21.15.191:3300/0 conn(0x7f11fc06df20 0x7f11fc158c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.359+0000 7f1200a00640 1 -- 172.21.15.28:0/2927039198 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc06eed0 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.359+0000 7f1201400640 1 --2- 172.21.15.28:0/2927039198 >> v2:172.21.15.191:3300/0 conn(0x7f11fc06df20 0x7f11fc158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.360+0000 7f1200a00640 1 --2- 172.21.15.28:0/2927039198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc0cb4c0 0x7f11fc155ce0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f11ec009870 tx=0x7f11ec02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf3093551fa6ba8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.361+0000 7f11fbe00640 1 -- 172.21.15.28:0/2927039198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec032440 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.361+0000 7f11fbe00640 1 -- 172.21.15.28:0/2927039198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ec0325a0 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.361+0000 7f11fbe00640 1 -- 172.21.15.28:0/2927039198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec032930 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.361+0000 7f1201e00640 1 -- 172.21.15.28:0/2927039198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc0cb4c0 msgr2=0x7f11fc155ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.361+0000 7f1201e00640 1 --2- 172.21.15.28:0/2927039198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc0cb4c0 0x7f11fc155ce0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f11ec009870 tx=0x7f11ec02a2a0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.361+0000 7f1201e00640 1 -- 172.21.15.28:0/2927039198 shutdown_connections 2024-09-13T16:44:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.361+0000 7f1201e00640 1 --2- 172.21.15.28:0/2927039198 >> v2:172.21.15.191:3300/0 conn(0x7f11fc06df20 0x7f11fc158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.361+0000 7f1201e00640 1 --2- 172.21.15.28:0/2927039198 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11fc0cb8a0 0x7f11fc156220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.361+0000 7f1201e00640 1 --2- 172.21.15.28:0/2927039198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc0cb4c0 0x7f11fc155ce0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.361+0000 7f1201e00640 1 -- 172.21.15.28:0/2927039198 >> 172.21.15.28:0/2927039198 conn(0x7f11fc065c50 msgr2=0x7f11fc066030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.361+0000 7f1201e00640 1 -- 172.21.15.28:0/2927039198 shutdown_connections 2024-09-13T16:44:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.361+0000 7f1201e00640 1 -- 172.21.15.28:0/2927039198 wait complete. 2024-09-13T16:44:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.363+0000 7f1201e00640 1 Processor -- start 2024-09-13T16:44:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.363+0000 7f1201e00640 1 -- start start 2024-09-13T16:44:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.364+0000 7f1201e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11fc06df20 0x7f11fc150be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.364+0000 7f1201e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc0cb4c0 0x7f11fc0bbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.364+0000 7f1201e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11fc0cb8a0 0x7f11fc0c1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.364+0000 7f1201e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc1ef080 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.364+0000 7f1201e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc1ef510 con 0x7f11fc06df20 2024-09-13T16:44:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.364+0000 7f1201e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc0c01e0 con 0x7f11fc0cb8a0 2024-09-13T16:44:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.364+0000 7f11f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc0cb4c0 0x7f11fc0bbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.364+0000 7f11f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc0cb4c0 0x7f11fc0bbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53636/0 (socket says 172.21.15.28:53636) 2024-09-13T16:44:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.364+0000 7f11f3e00640 1 -- 172.21.15.28:0/1655862393 learned_addr learned my addr 172.21.15.28:0/1655862393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.364+0000 7f1200a00640 1 --2- 172.21.15.28:0/1655862393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11fc06df20 0x7f11fc150be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.364+0000 7f1201400640 1 --2- 172.21.15.28:0/1655862393 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11fc0cb8a0 0x7f11fc0c1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.365+0000 7f11f3e00640 1 -- 172.21.15.28:0/1655862393 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11fc0cb8a0 msgr2=0x7f11fc0c1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.365+0000 7f11f3e00640 1 --2- 172.21.15.28:0/1655862393 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11fc0cb8a0 0x7f11fc0c1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.365+0000 7f11f3e00640 1 -- 172.21.15.28:0/1655862393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11fc06df20 msgr2=0x7f11fc150be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.365+0000 7f11f3e00640 1 --2- 172.21.15.28:0/1655862393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11fc06df20 0x7f11fc150be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.365+0000 7f11f3e00640 1 -- 172.21.15.28:0/1655862393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f4009a70 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.365+0000 7f1200a00640 1 --2- 172.21.15.28:0/1655862393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11fc06df20 0x7f11fc150be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.365+0000 7f11f3e00640 1 --2- 172.21.15.28:0/1655862393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc0cb4c0 0x7f11fc0bbdb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f11e400e980 tx=0x7f11e400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.365+0000 7f11f9600640 1 -- 172.21.15.28:0/1655862393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e4009670 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.365+0000 7f11f9600640 1 -- 172.21.15.28:0/1655862393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e400ca00 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.365+0000 7f11f9600640 1 -- 172.21.15.28:0/1655862393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e400cd90 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.365+0000 7f1201e00640 1 -- 172.21.15.28:0/1655862393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11ec0094d0 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.365+0000 7f1201e00640 1 -- 172.21.15.28:0/1655862393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11fc1eea30 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.367+0000 7f1201e00640 1 -- 172.21.15.28:0/1655862393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11fc1ef1f0 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.367+0000 7f11f9600640 1 -- 172.21.15.28:0/1655862393 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f11e4013470 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.368+0000 7f11f9600640 1 --2- 172.21.15.28:0/1655862393 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f11d806ac30 0x7f11d806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.368+0000 7f11f9600640 1 -- 172.21.15.28:0/1655862393 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11e40896e0 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.369+0000 7f1200a00640 1 --2- 172.21.15.28:0/1655862393 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f11d806ac30 0x7f11d806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:22.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.369+0000 7f1200a00640 1 --2- 172.21.15.28:0/1655862393 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f11d806ac30 0x7f11d806d0f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f11ec034000 tx=0x7f11ec02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.373+0000 7f11f9600640 1 -- 172.21.15.28:0/1655862393 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f11e4054a00 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.586+0000 7f1201e00640 1 -- 172.21.15.28:0/1655862393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11fc0c5c00 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.587+0000 7f11f9600640 1 -- 172.21.15.28:0/1655862393 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6308 (secure 0 0 0) 0x7f11e4058020 con 0x7f11fc0cb4c0 2024-09-13T16:44:22.588 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:22.589 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4227,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/355570369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":355570369},{"type":"v1","addr":"172.21.15.28:6835","nonce":355570369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:43:54.969699+0000","modified":"2024-09-13T16:44:20.667541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4250,"mds_2":4244,"mds_3":4235,"mds_4":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"f","rank":4,"incarnation":16,"state":"up:creating","state_seq":1,"addr":"172.21.15.191:6839/2366383263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6837","nonce":2366383263},{"type":"v1","addr":"172.21.15.191:6839","nonce":2366383263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4235":{"gid":4235,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.191:6838/2702860767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2702860767},{"type":"v1","addr":"172.21.15.191:6838","nonce":2702860767}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/2240589617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2240589617},{"type":"v1","addr":"172.21.15.28:6833","nonce":2240589617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6837/4068563017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":4068563017},{"type":"v1","addr":"172.21.15.28:6837","nonce":4068563017}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4258":{"gid":4258,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/704943585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":704943585},{"type":"v1","addr":"172.21.15.191:6835","nonce":704943585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T16:44:22.589 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 16 2024-09-13T16:44:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.589+0000 7f1201e00640 1 -- 172.21.15.28:0/1655862393 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f11d806ac30 msgr2=0x7f11d806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.589+0000 7f1201e00640 1 --2- 172.21.15.28:0/1655862393 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f11d806ac30 0x7f11d806d0f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f11ec034000 tx=0x7f11ec02e000 comp rx=0 tx=0).stop 2024-09-13T16:44:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.589+0000 7f1201e00640 1 -- 172.21.15.28:0/1655862393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc0cb4c0 msgr2=0x7f11fc0bbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.589+0000 7f1201e00640 1 --2- 172.21.15.28:0/1655862393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc0cb4c0 0x7f11fc0bbdb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f11e400e980 tx=0x7f11e400ee40 comp rx=0 tx=0).stop 2024-09-13T16:44:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.589+0000 7f1201e00640 1 -- 172.21.15.28:0/1655862393 shutdown_connections 2024-09-13T16:44:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.589+0000 7f1201e00640 1 --2- 172.21.15.28:0/1655862393 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f11d806ac30 0x7f11d806d0f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.589+0000 7f1201e00640 1 --2- 172.21.15.28:0/1655862393 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11fc0cb8a0 0x7f11fc0c1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.589+0000 7f1201e00640 1 --2- 172.21.15.28:0/1655862393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11fc0cb4c0 0x7f11fc0bbdb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.589+0000 7f1201e00640 1 --2- 172.21.15.28:0/1655862393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11fc06df20 0x7f11fc150be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.589+0000 7f1201e00640 1 -- 172.21.15.28:0/1655862393 >> 172.21.15.28:0/1655862393 conn(0x7f11fc065c50 msgr2=0x7f11fc0ca920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:22.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.589+0000 7f1201e00640 1 -- 172.21.15.28:0/1655862393 shutdown_connections 2024-09-13T16:44:22.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.589+0000 7f1201e00640 1 -- 172.21.15.28:0/1655862393 wait complete. 2024-09-13T16:44:22.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:44:22.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.760+0000 7fd34be00640 1 Processor -- start 2024-09-13T16:44:22.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.760+0000 7fd34be00640 1 -- start start 2024-09-13T16:44:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.761+0000 7fd34be00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3440c5bd0 0x7fd344156150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.761+0000 7fd34be00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd3440c6ef0 0x7fd3440c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.761+0000 7fd34be00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34414ca00 con 0x7fd3440c3670 2024-09-13T16:44:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.761+0000 7fd34be00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3440c2dc0 con 0x7fd3440c6ef0 2024-09-13T16:44:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.761+0000 7fd34be00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd344155780 con 0x7fd3440c5bd0 2024-09-13T16:44:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.762+0000 7fd349400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3440c5bd0 0x7fd344156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.762+0000 7fd349e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd3440c6ef0 0x7fd3440c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.762+0000 7fd348a00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fd3440c3670 0x7fd3440c1960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:47398/0 (socket says 172.21.15.28:47398) 2024-09-13T16:44:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.762+0000 7fd348a00640 1 -- 172.21.15.28:0/2391257497 learned_addr learned my addr 172.21.15.28:0/2391257497 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.762+0000 7fd349e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd3440c6ef0 0x7fd3440c7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:51038/0 (socket says 172.21.15.28:51038) 2024-09-13T16:44:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.762+0000 7fd349400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3440c5bd0 0x7fd344156150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:60462/0 (socket says 172.21.15.28:60462) 2024-09-13T16:44:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.762+0000 7fd349e00640 1 -- 172.21.15.28:0/2391257497 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3440c5bd0 msgr2=0x7fd344156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.762+0000 7fd349e00640 1 --2- 172.21.15.28:0/2391257497 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3440c5bd0 0x7fd344156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.762+0000 7fd349e00640 1 -- 172.21.15.28:0/2391257497 >> v1:172.21.15.191:6789/0 conn(0x7fd3440c3670 legacy=0x7fd3440c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.762+0000 7fd349e00640 1 -- 172.21.15.28:0/2391257497 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd344155990 con 0x7fd3440c6ef0 2024-09-13T16:44:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.762+0000 7fd343e00640 1 -- 172.21.15.28:0/2391257497 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156012107 0 0) 0x7fd34414ca00 con 0x7fd3440c3670 2024-09-13T16:44:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.763+0000 7fd349e00640 1 --2- 172.21.15.28:0/2391257497 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3440c6ef0 0x7fd3440c7370 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00bfa0 tx=0x7fd33c02c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40d2035e346f5c42 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.764+0000 7fd343e00640 1 -- 172.21.15.28:0/2391257497 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c009540 con 0x7fd3440c6ef0 2024-09-13T16:44:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.764+0000 7fd343e00640 1 -- 172.21.15.28:0/2391257497 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33c0096a0 con 0x7fd3440c6ef0 2024-09-13T16:44:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.764+0000 7fd343e00640 1 -- 172.21.15.28:0/2391257497 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c009a30 con 0x7fd3440c6ef0 2024-09-13T16:44:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.764+0000 7fd34be00640 1 -- 172.21.15.28:0/2391257497 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3440c6ef0 msgr2=0x7fd3440c7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.764+0000 7fd34be00640 1 --2- 172.21.15.28:0/2391257497 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3440c6ef0 0x7fd3440c7370 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00bfa0 tx=0x7fd33c02c8f0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.764+0000 7fd34be00640 1 -- 172.21.15.28:0/2391257497 shutdown_connections 2024-09-13T16:44:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.764+0000 7fd34be00640 1 --2- 172.21.15.28:0/2391257497 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3440c6ef0 0x7fd3440c7370 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.764+0000 7fd34be00640 1 --2- 172.21.15.28:0/2391257497 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3440c5bd0 0x7fd344156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.764+0000 7fd34be00640 1 -- 172.21.15.28:0/2391257497 >> 172.21.15.28:0/2391257497 conn(0x7fd344066040 msgr2=0x7fd344069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.764+0000 7fd34be00640 1 -- 172.21.15.28:0/2391257497 shutdown_connections 2024-09-13T16:44:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.764+0000 7fd34be00640 1 -- 172.21.15.28:0/2391257497 wait complete. 2024-09-13T16:44:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.766+0000 7fd34be00640 1 Processor -- start 2024-09-13T16:44:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.766+0000 7fd34be00640 1 -- start start 2024-09-13T16:44:22.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.767+0000 7fd34be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3440c3670 0x7fd34414d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:22.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.767+0000 7fd34be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3440c5bd0 0x7fd3440c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:22.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.767+0000 7fd34be00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3440c6ef0 0x7fd3440be460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:22.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.767+0000 7fd34be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34414d7e0 con 0x7fd3440c5bd0 2024-09-13T16:44:22.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.767+0000 7fd34be00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3440bfaa0 con 0x7fd3440c3670 2024-09-13T16:44:22.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.767+0000 7fd34be00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34414f860 con 0x7fd3440c6ef0 2024-09-13T16:44:22.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.767+0000 7fd348a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3440c5bd0 0x7fd3440c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:22.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.767+0000 7fd348a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3440c5bd0 0x7fd3440c8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53648/0 (socket says 172.21.15.28:53648) 2024-09-13T16:44:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.767+0000 7fd348a00640 1 -- 172.21.15.28:0/4114407909 learned_addr learned my addr 172.21.15.28:0/4114407909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.767+0000 7fd349400640 1 --2- 172.21.15.28:0/4114407909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3440c3670 0x7fd34414d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.767+0000 7fd349e00640 1 --2- 172.21.15.28:0/4114407909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3440c6ef0 0x7fd3440be460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.767+0000 7fd348a00640 1 -- 172.21.15.28:0/4114407909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3440c6ef0 msgr2=0x7fd3440be460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.767+0000 7fd348a00640 1 --2- 172.21.15.28:0/4114407909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3440c6ef0 0x7fd3440be460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.767+0000 7fd348a00640 1 -- 172.21.15.28:0/4114407909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3440c3670 msgr2=0x7fd34414d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.767+0000 7fd348a00640 1 --2- 172.21.15.28:0/4114407909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3440c3670 0x7fd34414d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.767+0000 7fd348a00640 1 -- 172.21.15.28:0/4114407909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c00bc80 con 0x7fd3440c5bd0 2024-09-13T16:44:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.768+0000 7fd348a00640 1 --2- 172.21.15.28:0/4114407909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3440c5bd0 0x7fd3440c8b30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd33000c4e0 tx=0x7fd33000c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.768+0000 7fd341600640 1 -- 172.21.15.28:0/4114407909 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3300074e0 con 0x7fd3440c5bd0 2024-09-13T16:44:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.768+0000 7fd341600640 1 -- 172.21.15.28:0/4114407909 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd330007660 con 0x7fd3440c5bd0 2024-09-13T16:44:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.768+0000 7fd341600640 1 -- 172.21.15.28:0/4114407909 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd330004a20 con 0x7fd3440c5bd0 2024-09-13T16:44:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.768+0000 7fd34be00640 1 -- 172.21.15.28:0/4114407909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34414edb0 con 0x7fd3440c5bd0 2024-09-13T16:44:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.768+0000 7fd34be00640 1 -- 172.21.15.28:0/4114407909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34414cfc0 con 0x7fd3440c5bd0 2024-09-13T16:44:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.769+0000 7fd34be00640 1 -- 172.21.15.28:0/4114407909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34414f9d0 con 0x7fd3440c5bd0 2024-09-13T16:44:22.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.770+0000 7fd341600640 1 -- 172.21.15.28:0/4114407909 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd330004b80 con 0x7fd3440c5bd0 2024-09-13T16:44:22.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.771+0000 7fd341600640 1 --2- 172.21.15.28:0/4114407909 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd32406ac00 0x7fd32406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:22.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.771+0000 7fd341600640 1 -- 172.21.15.28:0/4114407909 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd330088670 con 0x7fd3440c5bd0 2024-09-13T16:44:22.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.771+0000 7fd349400640 1 --2- 172.21.15.28:0/4114407909 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd32406ac00 0x7fd32406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.772+0000 7fd349400640 1 --2- 172.21.15.28:0/4114407909 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd32406ac00 0x7fd32406d0c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd338009b40 tx=0x7fd338009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:22.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.775+0000 7fd341600640 1 -- 172.21.15.28:0/4114407909 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd330053890 con 0x7fd3440c5bd0 2024-09-13T16:44:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.956+0000 7fd34be00640 1 -- 172.21.15.28:0/4114407909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd344155c30 con 0x7fd3440c5bd0 2024-09-13T16:44:22.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.957+0000 7fd341600640 1 -- 172.21.15.28:0/4114407909 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16481 (secure 0 0 0) 0x7fd330018020 con 0x7fd3440c5bd0 2024-09-13T16:44:22.959 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:22.959 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:43:55.971358+0000","last_up_change":"2024-09-13T16:43:49.476313+0000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:43:49.643606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:43:52.828510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:43:53.945874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6801","nonce":2964323435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6803","nonce":2964323435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6807","nonce":2964323435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6805","nonce":2964323435}]},"public_addr":"172.21.15.28:6801/2964323435","cluster_addr":"172.21.15.28:6803/2964323435","heartbeat_back_addr":"172.21.15.28:6807/2964323435","heartbeat_front_addr":"172.21.15.28:6805/2964323435","state":["exists","up"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6809","nonce":23768570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6811","nonce":23768570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6815","nonce":23768570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6813","nonce":23768570}]},"public_addr":"172.21.15.28:6809/23768570","cluster_addr":"172.21.15.28:6811/23768570","heartbeat_back_addr":"172.21.15.28:6815/23768570","heartbeat_front_addr":"172.21.15.28:6813/23768570","state":["exists","up"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6817","nonce":886887303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6819","nonce":886887303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6823","nonce":886887303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6821","nonce":886887303}]},"public_addr":"172.21.15.28:6817/886887303","cluster_addr":"172.21.15.28:6819/886887303","heartbeat_back_addr":"172.21.15.28:6823/886887303","heartbeat_front_addr":"172.21.15.28:6821/886887303","state":["exists","up"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6825","nonce":2616036674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6827","nonce":2616036674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6831","nonce":2616036674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6829","nonce":2616036674}]},"public_addr":"172.21.15.28:6825/2616036674","cluster_addr":"172.21.15.28:6827/2616036674","heartbeat_back_addr":"172.21.15.28:6831/2616036674","heartbeat_front_addr":"172.21.15.28:6829/2616036674","state":["exists","up"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6801","nonce":2817102450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6803","nonce":2817102450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6807","nonce":2817102450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6805","nonce":2817102450}]},"public_addr":"172.21.15.191:6801/2817102450","cluster_addr":"172.21.15.191:6803/2817102450","heartbeat_back_addr":"172.21.15.191:6807/2817102450","heartbeat_front_addr":"172.21.15.191:6805/2817102450","state":["exists","up"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6817","nonce":2503027971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6819","nonce":2503027971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6823","nonce":2503027971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6821","nonce":2503027971}]},"public_addr":"172.21.15.191:6817/2503027971","cluster_addr":"172.21.15.191:6819/2503027971","heartbeat_back_addr":"172.21.15.191:6823/2503027971","heartbeat_front_addr":"172.21.15.191:6821/2503027971","state":["exists","up"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6809","nonce":3006595088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6811","nonce":3006595088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6815","nonce":3006595088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6813","nonce":3006595088}]},"public_addr":"172.21.15.191:6809/3006595088","cluster_addr":"172.21.15.191:6811/3006595088","heartbeat_back_addr":"172.21.15.191:6815/3006595088","heartbeat_front_addr":"172.21.15.191:6813/3006595088","state":["exists","up"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6825","nonce":1477327720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6827","nonce":1477327720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6831","nonce":1477327720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6829","nonce":1477327720}]},"public_addr":"172.21.15.191:6825/1477327720","cluster_addr":"172.21.15.191:6827/1477327720","heartbeat_back_addr":"172.21.15.191:6831/1477327720","heartbeat_front_addr":"172.21.15.191:6829/1477327720","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.917250+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.954351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.017329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.046221+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.707467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.726947+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.782321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.764850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:44:22.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.959+0000 7fd34be00640 1 -- 172.21.15.28:0/4114407909 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd32406ac00 msgr2=0x7fd32406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:22.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.959+0000 7fd34be00640 1 --2- 172.21.15.28:0/4114407909 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd32406ac00 0x7fd32406d0c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd338009b40 tx=0x7fd338009190 comp rx=0 tx=0).stop 2024-09-13T16:44:22.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.959+0000 7fd34be00640 1 -- 172.21.15.28:0/4114407909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3440c5bd0 msgr2=0x7fd3440c8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:22.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.959+0000 7fd34be00640 1 --2- 172.21.15.28:0/4114407909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3440c5bd0 0x7fd3440c8b30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd33000c4e0 tx=0x7fd33000c9a0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.959+0000 7fd34be00640 1 -- 172.21.15.28:0/4114407909 shutdown_connections 2024-09-13T16:44:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.959+0000 7fd34be00640 1 --2- 172.21.15.28:0/4114407909 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd32406ac00 0x7fd32406d0c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.959+0000 7fd34be00640 1 --2- 172.21.15.28:0/4114407909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3440c6ef0 0x7fd3440be460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.959+0000 7fd34be00640 1 --2- 172.21.15.28:0/4114407909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3440c5bd0 0x7fd3440c8b30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.959+0000 7fd34be00640 1 --2- 172.21.15.28:0/4114407909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3440c3670 0x7fd34414d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.959+0000 7fd34be00640 1 -- 172.21.15.28:0/4114407909 >> 172.21.15.28:0/4114407909 conn(0x7fd344066040 msgr2=0x7fd34414b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.960+0000 7fd34be00640 1 -- 172.21.15.28:0/4114407909 shutdown_connections 2024-09-13T16:44:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:22.960+0000 7fd34be00640 1 -- 172.21.15.28:0/4114407909 wait complete. 2024-09-13T16:44:22.977 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:43:49.643606+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:43:52.828510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:43:53.945874+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:44:22.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:44:23.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.133+0000 7f1751a00640 1 Processor -- start 2024-09-13T16:44:23.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.133+0000 7f1751a00640 1 -- start start 2024-09-13T16:44:23.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.134+0000 7f1751a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f174c14fd80 0x7f174c0c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:23.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.134+0000 7f1751a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f174c0c44e0 0x7f174c0c9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:23.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.134+0000 7f1751a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f174c1567d0 con 0x7f174c14f9a0 2024-09-13T16:44:23.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.134+0000 7f1751a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f174c0c2580 con 0x7f174c14fd80 2024-09-13T16:44:23.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.134+0000 7f1751a00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f174c155800 con 0x7f174c0c44e0 2024-09-13T16:44:23.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.135+0000 7f1742000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f174c14fd80 0x7f174c0c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:23.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.135+0000 7f174aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f174c14f9a0 0x7f174c0c5570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60988/0 (socket says 172.21.15.28:60988) 2024-09-13T16:44:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.135+0000 7f174aa00640 1 -- 172.21.15.28:0/1044621244 learned_addr learned my addr 172.21.15.28:0/1044621244 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.135+0000 7f174b400640 1 --2- 172.21.15.28:0/1044621244 >> v2:172.21.15.191:3300/0 conn(0x7f174c0c44e0 0x7f174c0c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.135+0000 7f1742000640 1 -- 172.21.15.28:0/1044621244 >> v2:172.21.15.191:3300/0 conn(0x7f174c0c44e0 msgr2=0x7f174c0c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.135+0000 7f1742000640 1 --2- 172.21.15.28:0/1044621244 >> v2:172.21.15.191:3300/0 conn(0x7f174c0c44e0 0x7f174c0c9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.135+0000 7f1742000640 1 -- 172.21.15.28:0/1044621244 >> v1:172.21.15.28:6789/0 conn(0x7f174c14f9a0 legacy=0x7f174c0c5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.135+0000 7f1742000640 1 -- 172.21.15.28:0/1044621244 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c0c1ef0 con 0x7f174c14fd80 2024-09-13T16:44:23.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.135+0000 7f174a000640 1 -- 172.21.15.28:0/1044621244 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528492114 0 0) 0x7f174c1567d0 con 0x7f174c14f9a0 2024-09-13T16:44:23.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.136+0000 7f1742000640 1 --2- 172.21.15.28:0/1044621244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f174c14fd80 0x7f174c0c9300 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f174c073840 tx=0x7f173402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f56eb77699f8f06 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.136+0000 7f174a000640 1 -- 172.21.15.28:0/1044621244 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1734032440 con 0x7f174c14fd80 2024-09-13T16:44:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.136+0000 7f174a000640 1 -- 172.21.15.28:0/1044621244 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17340325a0 con 0x7f174c14fd80 2024-09-13T16:44:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.136+0000 7f174a000640 1 -- 172.21.15.28:0/1044621244 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1734032930 con 0x7f174c14fd80 2024-09-13T16:44:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.137+0000 7f1751a00640 1 -- 172.21.15.28:0/1044621244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f174c14fd80 msgr2=0x7f174c0c9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.137+0000 7f1751a00640 1 --2- 172.21.15.28:0/1044621244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f174c14fd80 0x7f174c0c9300 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f174c073840 tx=0x7f173402a230 comp rx=0 tx=0).stop 2024-09-13T16:44:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.137+0000 7f1751a00640 1 -- 172.21.15.28:0/1044621244 shutdown_connections 2024-09-13T16:44:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.137+0000 7f1751a00640 1 --2- 172.21.15.28:0/1044621244 >> v2:172.21.15.191:3300/0 conn(0x7f174c0c44e0 0x7f174c0c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.137+0000 7f1751a00640 1 --2- 172.21.15.28:0/1044621244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f174c14fd80 0x7f174c0c9300 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.137+0000 7f1751a00640 1 -- 172.21.15.28:0/1044621244 >> 172.21.15.28:0/1044621244 conn(0x7f174c0ca2f0 msgr2=0x7f174c0ca6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.137+0000 7f1751a00640 1 -- 172.21.15.28:0/1044621244 shutdown_connections 2024-09-13T16:44:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.137+0000 7f1751a00640 1 -- 172.21.15.28:0/1044621244 wait complete. 2024-09-13T16:44:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.139+0000 7f1751a00640 1 Processor -- start 2024-09-13T16:44:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.139+0000 7f1751a00640 1 -- start start 2024-09-13T16:44:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.140+0000 7f1751a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f174c0c44e0 0x7f174c158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.140+0000 7f1751a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f174c14f9a0 0x7f174c150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.140+0000 7f1751a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f174c151330 0x7f174c1e1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.140+0000 7f1751a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c1517c0 con 0x7f174c151330 2024-09-13T16:44:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.140+0000 7f1751a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c1deb70 con 0x7f174c0c44e0 2024-09-13T16:44:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.140+0000 7f1751a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174c06e630 con 0x7f174c14f9a0 2024-09-13T16:44:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.140+0000 7f174aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f174c0c44e0 0x7f174c158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.140+0000 7f174b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f174c151330 0x7f174c1e1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.140+0000 7f1742000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f174c14f9a0 0x7f174c150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.140+0000 7f174aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f174c0c44e0 0x7f174c158d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:51058/0 (socket says 172.21.15.28:51058) 2024-09-13T16:44:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.140+0000 7f174aa00640 1 -- 172.21.15.28:0/141566229 learned_addr learned my addr 172.21.15.28:0/141566229 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.140+0000 7f174b400640 1 -- 172.21.15.28:0/141566229 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f174c14f9a0 msgr2=0x7f174c150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.140+0000 7f174b400640 1 --2- 172.21.15.28:0/141566229 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f174c14f9a0 0x7f174c150df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.140+0000 7f174b400640 1 -- 172.21.15.28:0/141566229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f174c0c44e0 msgr2=0x7f174c158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.140+0000 7f174b400640 1 --2- 172.21.15.28:0/141566229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f174c0c44e0 0x7f174c158d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.140+0000 7f174b400640 1 -- 172.21.15.28:0/141566229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1734009400 con 0x7f174c151330 2024-09-13T16:44:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.141+0000 7f174b400640 1 --2- 172.21.15.28:0/141566229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f174c151330 0x7f174c1e1030 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f174400efc0 tx=0x7f174400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.141+0000 7f1743400640 1 -- 172.21.15.28:0/141566229 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744009c00 con 0x7f174c151330 2024-09-13T16:44:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.141+0000 7f1743400640 1 -- 172.21.15.28:0/141566229 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174400f040 con 0x7f174c151330 2024-09-13T16:44:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.141+0000 7f1743400640 1 -- 172.21.15.28:0/141566229 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17440146c0 con 0x7f174c151330 2024-09-13T16:44:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.141+0000 7f1751a00640 1 -- 172.21.15.28:0/141566229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f174c1ee570 con 0x7f174c151330 2024-09-13T16:44:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.141+0000 7f1751a00640 1 -- 172.21.15.28:0/141566229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f174c1505f0 con 0x7f174c151330 2024-09-13T16:44:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.142+0000 7f1751a00640 1 -- 172.21.15.28:0/141566229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f174c06f8f0 con 0x7f174c151330 2024-09-13T16:44:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.143+0000 7f1743400640 1 -- 172.21.15.28:0/141566229 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1744014820 con 0x7f174c151330 2024-09-13T16:44:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.144+0000 7f1743400640 1 --2- 172.21.15.28:0/141566229 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f172806ac00 0x7f172806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.144+0000 7f1743400640 1 -- 172.21.15.28:0/141566229 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f174408a0e0 con 0x7f174c151330 2024-09-13T16:44:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.144+0000 7f174aa00640 1 --2- 172.21.15.28:0/141566229 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f172806ac00 0x7f172806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.145+0000 7f174aa00640 1 --2- 172.21.15.28:0/141566229 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f172806ac00 0x7f172806d0c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f173c000f60 tx=0x7f173c007e10 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.148+0000 7f1743400640 1 -- 172.21.15.28:0/141566229 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1744055360 con 0x7f174c151330 2024-09-13T16:44:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.328+0000 7f1751a00640 1 -- 172.21.15.28:0/141566229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f174c150ab0 con 0x7f174c151330 2024-09-13T16:44:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.328+0000 7f1743400640 1 -- 172.21.15.28:0/141566229 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1744058980 con 0x7f174c151330 2024-09-13T16:44:23.330 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T16:44:23.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.330+0000 7f1751a00640 1 -- 172.21.15.28:0/141566229 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f172806ac00 msgr2=0x7f172806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.330+0000 7f1751a00640 1 --2- 172.21.15.28:0/141566229 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f172806ac00 0x7f172806d0c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f173c000f60 tx=0x7f173c007e10 comp rx=0 tx=0).stop 2024-09-13T16:44:23.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.330+0000 7f1751a00640 1 -- 172.21.15.28:0/141566229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f174c151330 msgr2=0x7f174c1e1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.330+0000 7f1751a00640 1 --2- 172.21.15.28:0/141566229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f174c151330 0x7f174c1e1030 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f174400efc0 tx=0x7f174400c490 comp rx=0 tx=0).stop 2024-09-13T16:44:23.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.330+0000 7f1751a00640 1 -- 172.21.15.28:0/141566229 shutdown_connections 2024-09-13T16:44:23.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.331+0000 7f1751a00640 1 --2- 172.21.15.28:0/141566229 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f172806ac00 0x7f172806d0c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.331+0000 7f1751a00640 1 --2- 172.21.15.28:0/141566229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f174c151330 0x7f174c1e1030 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.331+0000 7f1751a00640 1 --2- 172.21.15.28:0/141566229 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f174c14f9a0 0x7f174c150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.331+0000 7f1751a00640 1 --2- 172.21.15.28:0/141566229 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f174c0c44e0 0x7f174c158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.331+0000 7f1751a00640 1 -- 172.21.15.28:0/141566229 >> 172.21.15.28:0/141566229 conn(0x7f174c0ca2f0 msgr2=0x7f174c06cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:23.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.331+0000 7f1751a00640 1 -- 172.21.15.28:0/141566229 shutdown_connections 2024-09-13T16:44:23.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.331+0000 7f1751a00640 1 -- 172.21.15.28:0/141566229 wait complete. 2024-09-13T16:44:23.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:44:23.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.505+0000 7f3fcfe00640 1 Processor -- start 2024-09-13T16:44:23.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.505+0000 7f3fcfe00640 1 -- start start 2024-09-13T16:44:23.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.506+0000 7f3fcfe00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3fc80c73e0 0x7f3fc80c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:23.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.506+0000 7f3fcfe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc80c6f50 con 0x7f3fc80c5a60 2024-09-13T16:44:23.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.506+0000 7f3fcfe00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc80c6940 con 0x7f3fc80c5140 2024-09-13T16:44:23.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.506+0000 7f3fcfe00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fc80c1b50 con 0x7f3fc80c73e0 2024-09-13T16:44:23.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.506+0000 7f3fcde00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3fc80c5a60 0x7f3fc814f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60990/0 (socket says 172.21.15.28:60990) 2024-09-13T16:44:23.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.506+0000 7f3fcde00640 1 -- 172.21.15.28:0/392990001 learned_addr learned my addr 172.21.15.28:0/392990001 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:23.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.506+0000 7f3fcca00640 1 --2- 172.21.15.28:0/392990001 >> v2:172.21.15.191:3300/0 conn(0x7f3fc80c73e0 0x7f3fc80c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:23.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.507+0000 7f3fc7e00640 1 -- 172.21.15.28:0/392990001 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450328370 0 0) 0x7f3fc80c6f50 con 0x7f3fc80c5a60 2024-09-13T16:44:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.507+0000 7f3fc7e00640 1 -- 172.21.15.28:0/392990001 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fb0003520 con 0x7f3fc80c5a60 2024-09-13T16:44:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.507+0000 7f3fcca00640 1 -- 172.21.15.28:0/392990001 >> v1:172.21.15.191:6789/0 conn(0x7f3fc80c5140 legacy=0x7f3fc8159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.507+0000 7f3fcca00640 1 -- 172.21.15.28:0/392990001 >> v1:172.21.15.28:6789/0 conn(0x7f3fc80c5a60 legacy=0x7f3fc814f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.507+0000 7f3fcca00640 1 -- 172.21.15.28:0/392990001 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc80c1e30 con 0x7f3fc80c73e0 2024-09-13T16:44:23.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.508+0000 7f3fcca00640 1 --2- 172.21.15.28:0/392990001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fc80c73e0 0x7f3fc80c5520 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3fb40097a0 tx=0x7f3fb402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfd33f694c86c713 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:23.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.508+0000 7f3fc7e00640 1 -- 172.21.15.28:0/392990001 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4032440 con 0x7f3fc80c73e0 2024-09-13T16:44:23.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.508+0000 7f3fc7e00640 1 -- 172.21.15.28:0/392990001 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb40325a0 con 0x7f3fc80c73e0 2024-09-13T16:44:23.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.508+0000 7f3fc7e00640 1 -- 172.21.15.28:0/392990001 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4032910 con 0x7f3fc80c73e0 2024-09-13T16:44:23.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.509+0000 7f3fcfe00640 1 -- 172.21.15.28:0/392990001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fc80c73e0 msgr2=0x7f3fc80c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.509+0000 7f3fcfe00640 1 --2- 172.21.15.28:0/392990001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fc80c73e0 0x7f3fc80c5520 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3fb40097a0 tx=0x7f3fb402a1a0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.509+0000 7f3fcfe00640 1 -- 172.21.15.28:0/392990001 shutdown_connections 2024-09-13T16:44:23.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.509+0000 7f3fcfe00640 1 --2- 172.21.15.28:0/392990001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fc80c73e0 0x7f3fc80c5520 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.509+0000 7f3fcfe00640 1 -- 172.21.15.28:0/392990001 >> 172.21.15.28:0/392990001 conn(0x7f3fc80ca6b0 msgr2=0x7f3fc80caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:23.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.509+0000 7f3fcfe00640 1 -- 172.21.15.28:0/392990001 shutdown_connections 2024-09-13T16:44:23.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.509+0000 7f3fcfe00640 1 -- 172.21.15.28:0/392990001 wait complete. 2024-09-13T16:44:23.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.510+0000 7f3fcfe00640 1 Processor -- start 2024-09-13T16:44:23.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.510+0000 7f3fcfe00640 1 -- start start 2024-09-13T16:44:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.511+0000 7f3fcfe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fc80c5140 0x7f3fc81eed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.511+0000 7f3fcfe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fc80c5a60 0x7f3fc8148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.511+0000 7f3fcfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fc80c73e0 0x7f3fc8142220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.511+0000 7f3fcfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc8148bc0 con 0x7f3fc80c73e0 2024-09-13T16:44:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.511+0000 7f3fcfe00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc8146f00 con 0x7f3fc80c5140 2024-09-13T16:44:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.511+0000 7f3fcfe00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc81ef2b0 con 0x7f3fc80c5a60 2024-09-13T16:44:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.512+0000 7f3fcde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fc80c73e0 0x7f3fc8142220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.512+0000 7f3fcca00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fc80c5a60 0x7f3fc8148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.512+0000 7f3fcde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fc80c73e0 0x7f3fc8142220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53680/0 (socket says 172.21.15.28:53680) 2024-09-13T16:44:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.512+0000 7f3fcde00640 1 -- 172.21.15.28:0/3798851784 learned_addr learned my addr 172.21.15.28:0/3798851784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.512+0000 7f3fcd400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fc80c5140 0x7f3fc81eed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.512+0000 7f3fcde00640 1 -- 172.21.15.28:0/3798851784 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fc80c5a60 msgr2=0x7f3fc8148fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.512+0000 7f3fcde00640 1 --2- 172.21.15.28:0/3798851784 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fc80c5a60 0x7f3fc8148fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.512+0000 7f3fcde00640 1 -- 172.21.15.28:0/3798851784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fc80c5140 msgr2=0x7f3fc81eed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.512+0000 7f3fcde00640 1 --2- 172.21.15.28:0/3798851784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fc80c5140 0x7f3fc81eed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.512+0000 7f3fcde00640 1 -- 172.21.15.28:0/3798851784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb4009400 con 0x7f3fc80c73e0 2024-09-13T16:44:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.512+0000 7f3fcd400640 1 --2- 172.21.15.28:0/3798851784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fc80c5140 0x7f3fc81eed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.512+0000 7f3fcde00640 1 --2- 172.21.15.28:0/3798851784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fc80c73e0 0x7f3fc8142220 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3fc000cb70 tx=0x7f3fc000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.512+0000 7f3fc5600640 1 -- 172.21.15.28:0/3798851784 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc000ad40 con 0x7f3fc80c73e0 2024-09-13T16:44:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.512+0000 7f3fc5600640 1 -- 172.21.15.28:0/3798851784 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc0012400 con 0x7f3fc80c73e0 2024-09-13T16:44:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.512+0000 7f3fc5600640 1 -- 172.21.15.28:0/3798851784 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc000aea0 con 0x7f3fc80c73e0 2024-09-13T16:44:23.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.512+0000 7f3fcfe00640 1 -- 172.21.15.28:0/3798851784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc81467e0 con 0x7f3fc80c73e0 2024-09-13T16:44:23.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.513+0000 7f3fcfe00640 1 -- 172.21.15.28:0/3798851784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc8148220 con 0x7f3fc80c73e0 2024-09-13T16:44:23.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.514+0000 7f3fcfe00640 1 -- 172.21.15.28:0/3798851784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc8148d00 con 0x7f3fc80c73e0 2024-09-13T16:44:23.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.514+0000 7f3fc5600640 1 -- 172.21.15.28:0/3798851784 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3fc0007070 con 0x7f3fc80c73e0 2024-09-13T16:44:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.515+0000 7f3fc5600640 1 --2- 172.21.15.28:0/3798851784 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3fa806ac00 0x7f3fa806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.515+0000 7f3fc5600640 1 -- 172.21.15.28:0/3798851784 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3fc00892f0 con 0x7f3fc80c73e0 2024-09-13T16:44:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.516+0000 7f3fcd400640 1 --2- 172.21.15.28:0/3798851784 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3fa806ac00 0x7f3fa806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:23.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.516+0000 7f3fcd400640 1 --2- 172.21.15.28:0/3798851784 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3fa806ac00 0x7f3fa806d0c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc000f60 tx=0x7f3fbc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:23.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.519+0000 7f3fc5600640 1 -- 172.21.15.28:0/3798851784 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3fc8148d00 con 0x7f3fc80c73e0 2024-09-13T16:44:23.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.702+0000 7f3fcfe00640 1 -- 172.21.15.28:0/3798851784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3fc8148d00 con 0x7f3fc80c73e0 2024-09-13T16:44:23.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.702+0000 7f3fc5600640 1 -- 172.21.15.28:0/3798851784 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3fc8148d00 con 0x7f3fc80c73e0 2024-09-13T16:44:23.704 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T16:44:23.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.704+0000 7f3fcfe00640 1 -- 172.21.15.28:0/3798851784 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3fa806ac00 msgr2=0x7f3fa806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.704+0000 7f3fcfe00640 1 --2- 172.21.15.28:0/3798851784 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3fa806ac00 0x7f3fa806d0c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc000f60 tx=0x7f3fbc009040 comp rx=0 tx=0).stop 2024-09-13T16:44:23.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.704+0000 7f3fcfe00640 1 -- 172.21.15.28:0/3798851784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fc80c73e0 msgr2=0x7f3fc8142220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.704+0000 7f3fcfe00640 1 --2- 172.21.15.28:0/3798851784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fc80c73e0 0x7f3fc8142220 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3fc000cb70 tx=0x7f3fc000a430 comp rx=0 tx=0).stop 2024-09-13T16:44:23.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.705+0000 7f3fcfe00640 1 -- 172.21.15.28:0/3798851784 shutdown_connections 2024-09-13T16:44:23.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.705+0000 7f3fcfe00640 1 --2- 172.21.15.28:0/3798851784 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3fa806ac00 0x7f3fa806d0c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.705+0000 7f3fcfe00640 1 --2- 172.21.15.28:0/3798851784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fc80c73e0 0x7f3fc8142220 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.705+0000 7f3fcfe00640 1 --2- 172.21.15.28:0/3798851784 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fc80c5a60 0x7f3fc8148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.705+0000 7f3fcfe00640 1 --2- 172.21.15.28:0/3798851784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fc80c5140 0x7f3fc81eed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.705+0000 7f3fcfe00640 1 -- 172.21.15.28:0/3798851784 >> 172.21.15.28:0/3798851784 conn(0x7f3fc80ca6b0 msgr2=0x7f3fc806d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:23.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.705+0000 7f3fcfe00640 1 -- 172.21.15.28:0/3798851784 shutdown_connections 2024-09-13T16:44:23.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.705+0000 7f3fcfe00640 1 -- 172.21.15.28:0/3798851784 wait complete. 2024-09-13T16:44:23.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:44:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.878+0000 7f3a98e00640 1 Processor -- start 2024-09-13T16:44:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.879+0000 7f3a98e00640 1 -- start start 2024-09-13T16:44:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.880+0000 7f3a98e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a9406e6b0 0x7f3a941594f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.880+0000 7f3a98e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a9406b810 con 0x7f3a9406ea90 2024-09-13T16:44:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.880+0000 7f3a98e00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a94159a30 con 0x7f3a9406f1c0 2024-09-13T16:44:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.880+0000 7f3a98e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a941590d0 con 0x7f3a9406e6b0 2024-09-13T16:44:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.880+0000 7f3a93400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a9406e6b0 0x7f3a941594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.880+0000 7f3a8aa00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f3a9406f1c0 0x7f3a940c75a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:47424/0 (socket says 172.21.15.28:47424) 2024-09-13T16:44:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.880+0000 7f3a8aa00640 1 -- 172.21.15.28:0/1936588184 learned_addr learned my addr 172.21.15.28:0/1936588184 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.880+0000 7f3a93e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3a9406ea90 0x7f3a94155d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60996/0 (socket says 172.21.15.28:60996) 2024-09-13T16:44:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.881+0000 7f3a93400640 1 -- 172.21.15.28:0/1936588184 >> v1:172.21.15.191:6789/0 conn(0x7f3a9406f1c0 legacy=0x7f3a940c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.881+0000 7f3a93400640 1 -- 172.21.15.28:0/1936588184 >> v1:172.21.15.28:6789/0 conn(0x7f3a9406ea90 legacy=0x7f3a94155d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.881+0000 7f3a92a00640 1 -- 172.21.15.28:0/1936588184 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875901932 0 0) 0x7f3a94159a30 con 0x7f3a9406f1c0 2024-09-13T16:44:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.881+0000 7f3a93400640 1 -- 172.21.15.28:0/1936588184 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9406af90 con 0x7f3a9406e6b0 2024-09-13T16:44:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.882+0000 7f3a93400640 1 --2- 172.21.15.28:0/1936588184 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a9406e6b0 0x7f3a941594f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3a840097d0 tx=0x7f3a8402a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ce080acbabe1848 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:23.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.882+0000 7f3a92a00640 1 -- 172.21.15.28:0/1936588184 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8402ae30 con 0x7f3a9406e6b0 2024-09-13T16:44:23.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.882+0000 7f3a92a00640 1 -- 172.21.15.28:0/1936588184 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a84032440 con 0x7f3a9406e6b0 2024-09-13T16:44:23.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.882+0000 7f3a92a00640 1 -- 172.21.15.28:0/1936588184 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a840327d0 con 0x7f3a9406e6b0 2024-09-13T16:44:23.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.882+0000 7f3a98e00640 1 -- 172.21.15.28:0/1936588184 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a9406e6b0 msgr2=0x7f3a941594f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.882+0000 7f3a98e00640 1 --2- 172.21.15.28:0/1936588184 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a9406e6b0 0x7f3a941594f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3a840097d0 tx=0x7f3a8402a1a0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.883+0000 7f3a98e00640 1 -- 172.21.15.28:0/1936588184 shutdown_connections 2024-09-13T16:44:23.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.883+0000 7f3a98e00640 1 --2- 172.21.15.28:0/1936588184 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a9406e6b0 0x7f3a941594f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.883+0000 7f3a98e00640 1 -- 172.21.15.28:0/1936588184 >> 172.21.15.28:0/1936588184 conn(0x7f3a9406a2a0 msgr2=0x7f3a94065c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:23.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.883+0000 7f3a98e00640 1 -- 172.21.15.28:0/1936588184 shutdown_connections 2024-09-13T16:44:23.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.883+0000 7f3a98e00640 1 -- 172.21.15.28:0/1936588184 wait complete. 2024-09-13T16:44:23.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.884+0000 7f3a98e00640 1 Processor -- start 2024-09-13T16:44:23.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.884+0000 7f3a98e00640 1 -- start start 2024-09-13T16:44:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.885+0000 7f3a98e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a9406e6b0 0x7f3a940c39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.885+0000 7f3a98e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a9406ea90 0x7f3a94157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.885+0000 7f3a98e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9406f1c0 0x7f3a94157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.885+0000 7f3a98e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a94156b10 con 0x7f3a9406f1c0 2024-09-13T16:44:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.885+0000 7f3a98e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a941580c0 con 0x7f3a9406ea90 2024-09-13T16:44:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.885+0000 7f3a98e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a940c3f00 con 0x7f3a9406e6b0 2024-09-13T16:44:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.885+0000 7f3a93e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9406f1c0 0x7f3a94157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.885+0000 7f3a93e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9406f1c0 0x7f3a94157b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53696/0 (socket says 172.21.15.28:53696) 2024-09-13T16:44:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.885+0000 7f3a93e00640 1 -- 172.21.15.28:0/422586909 learned_addr learned my addr 172.21.15.28:0/422586909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.885+0000 7f3a93400640 1 --2- 172.21.15.28:0/422586909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a9406e6b0 0x7f3a940c39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.885+0000 7f3a8aa00640 1 --2- 172.21.15.28:0/422586909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a9406ea90 0x7f3a94157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.886+0000 7f3a93e00640 1 -- 172.21.15.28:0/422586909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a9406e6b0 msgr2=0x7f3a940c39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.886+0000 7f3a93e00640 1 --2- 172.21.15.28:0/422586909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a9406e6b0 0x7f3a940c39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.886+0000 7f3a93e00640 1 -- 172.21.15.28:0/422586909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a9406ea90 msgr2=0x7f3a94157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.886+0000 7f3a93e00640 1 --2- 172.21.15.28:0/422586909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a9406ea90 0x7f3a94157640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.886+0000 7f3a93e00640 1 -- 172.21.15.28:0/422586909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a84009400 con 0x7f3a9406f1c0 2024-09-13T16:44:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.886+0000 7f3a93e00640 1 --2- 172.21.15.28:0/422586909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9406f1c0 0x7f3a94157b80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00cb20 tx=0x7f3a8c002080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.886+0000 7f3a8be00640 1 -- 172.21.15.28:0/422586909 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c00a6a0 con 0x7f3a9406f1c0 2024-09-13T16:44:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.886+0000 7f3a8be00640 1 -- 172.21.15.28:0/422586909 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8c00a800 con 0x7f3a9406f1c0 2024-09-13T16:44:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.886+0000 7f3a8be00640 1 -- 172.21.15.28:0/422586909 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c00ab90 con 0x7f3a9406f1c0 2024-09-13T16:44:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.886+0000 7f3a98e00640 1 -- 172.21.15.28:0/422586909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a940b06b0 con 0x7f3a9406f1c0 2024-09-13T16:44:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.887+0000 7f3a98e00640 1 -- 172.21.15.28:0/422586909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a941ef6b0 con 0x7f3a9406f1c0 2024-09-13T16:44:23.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.888+0000 7f3a98e00640 1 -- 172.21.15.28:0/422586909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a940c84c0 con 0x7f3a9406f1c0 2024-09-13T16:44:23.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.888+0000 7f3a8be00640 1 -- 172.21.15.28:0/422586909 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3a8c00acf0 con 0x7f3a9406f1c0 2024-09-13T16:44:23.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.889+0000 7f3a8be00640 1 --2- 172.21.15.28:0/422586909 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3a7006ac00 0x7f3a7006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.889+0000 7f3a8be00640 1 -- 172.21.15.28:0/422586909 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a8c088b10 con 0x7f3a9406f1c0 2024-09-13T16:44:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.889+0000 7f3a93400640 1 --2- 172.21.15.28:0/422586909 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3a7006ac00 0x7f3a7006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.890+0000 7f3a93400640 1 --2- 172.21.15.28:0/422586909 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3a7006ac00 0x7f3a7006d0c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3a84034000 tx=0x7f3a8402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:23.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:23.893+0000 7f3a8be00640 1 -- 172.21.15.28:0/422586909 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3a8c053c70 con 0x7f3a9406f1c0 2024-09-13T16:44:24.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.070+0000 7f3a98e00640 1 -- 172.21.15.28:0/422586909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3a940673d0 con 0x7f3a9406f1c0 2024-09-13T16:44:24.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.071+0000 7f3a8be00640 1 -- 172.21.15.28:0/422586909 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3a8c018030 con 0x7f3a9406f1c0 2024-09-13T16:44:24.072 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T16:44:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.073+0000 7f3a98e00640 1 -- 172.21.15.28:0/422586909 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3a7006ac00 msgr2=0x7f3a7006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.073+0000 7f3a98e00640 1 --2- 172.21.15.28:0/422586909 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3a7006ac00 0x7f3a7006d0c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3a84034000 tx=0x7f3a8402e000 comp rx=0 tx=0).stop 2024-09-13T16:44:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.073+0000 7f3a98e00640 1 -- 172.21.15.28:0/422586909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9406f1c0 msgr2=0x7f3a94157b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.073+0000 7f3a98e00640 1 --2- 172.21.15.28:0/422586909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9406f1c0 0x7f3a94157b80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00cb20 tx=0x7f3a8c002080 comp rx=0 tx=0).stop 2024-09-13T16:44:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.073+0000 7f3a98e00640 1 -- 172.21.15.28:0/422586909 shutdown_connections 2024-09-13T16:44:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.073+0000 7f3a98e00640 1 --2- 172.21.15.28:0/422586909 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3a7006ac00 0x7f3a7006d0c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.073+0000 7f3a98e00640 1 --2- 172.21.15.28:0/422586909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9406f1c0 0x7f3a94157b80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.073+0000 7f3a98e00640 1 --2- 172.21.15.28:0/422586909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a9406ea90 0x7f3a94157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.073+0000 7f3a98e00640 1 --2- 172.21.15.28:0/422586909 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a9406e6b0 0x7f3a940c39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.073+0000 7f3a98e00640 1 -- 172.21.15.28:0/422586909 >> 172.21.15.28:0/422586909 conn(0x7f3a9406a2a0 msgr2=0x7f3a940c95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.073+0000 7f3a98e00640 1 -- 172.21.15.28:0/422586909 shutdown_connections 2024-09-13T16:44:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.073+0000 7f3a98e00640 1 -- 172.21.15.28:0/422586909 wait complete. 2024-09-13T16:44:24.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:44:24.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.242+0000 7ff1a0c00640 1 Processor -- start 2024-09-13T16:44:24.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.242+0000 7ff1a0c00640 1 -- start start 2024-09-13T16:44:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.244+0000 7ff1a0c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff19c1568e0 0x7ff19c0c2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.244+0000 7ff1a0c00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff19c0c4d60 0x7ff19c0c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.244+0000 7ff1a0c00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c151080 con 0x7ff19c150ca0 2024-09-13T16:44:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.244+0000 7ff1a0c00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff19c15ae40 con 0x7ff19c0c4d60 2024-09-13T16:44:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.244+0000 7ff1a0c00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c06a990 con 0x7ff19c1568e0 2024-09-13T16:44:24.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.244+0000 7ff199e00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7ff19c150ca0 0x7ff19c15a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:47434/0 (socket says 172.21.15.28:47434) 2024-09-13T16:44:24.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.244+0000 7ff199e00640 1 -- 172.21.15.28:0/3964069847 learned_addr learned my addr 172.21.15.28:0/3964069847 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:24.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.244+0000 7ff191400640 1 --2- 172.21.15.28:0/3964069847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff19c1568e0 0x7ff19c0c2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:24.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.244+0000 7ff19a800640 1 --2- 172.21.15.28:0/3964069847 >> v2:172.21.15.191:3300/0 conn(0x7ff19c0c4d60 0x7ff19c0c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:24.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.244+0000 7ff19a800640 1 -- 172.21.15.28:0/3964069847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff19c1568e0 msgr2=0x7ff19c0c2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:24.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.244+0000 7ff19a800640 1 --2- 172.21.15.28:0/3964069847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff19c1568e0 0x7ff19c0c2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.244+0000 7ff19a800640 1 -- 172.21.15.28:0/3964069847 >> v1:172.21.15.191:6789/0 conn(0x7ff19c150ca0 legacy=0x7ff19c15a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:24.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.244+0000 7ff19a800640 1 -- 172.21.15.28:0/3964069847 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c0c2870 con 0x7ff19c0c4d60 2024-09-13T16:44:24.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.244+0000 7ff199400640 1 -- 172.21.15.28:0/3964069847 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807228681 0 0) 0x7ff19c151080 con 0x7ff19c150ca0 2024-09-13T16:44:24.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.245+0000 7ff19a800640 1 --2- 172.21.15.28:0/3964069847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff19c0c4d60 0x7ff19c0c51f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff194009d90 tx=0x7ff19402a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3d6d46b8acfc2f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:24.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.245+0000 7ff199400640 1 -- 172.21.15.28:0/3964069847 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff194031600 con 0x7ff19c0c4d60 2024-09-13T16:44:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.246+0000 7ff199400640 1 -- 172.21.15.28:0/3964069847 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19402d050 con 0x7ff19c0c4d60 2024-09-13T16:44:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.246+0000 7ff199400640 1 -- 172.21.15.28:0/3964069847 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff194031980 con 0x7ff19c0c4d60 2024-09-13T16:44:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.246+0000 7ff1a0c00640 1 -- 172.21.15.28:0/3964069847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff19c0c4d60 msgr2=0x7ff19c0c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.246+0000 7ff1a0c00640 1 --2- 172.21.15.28:0/3964069847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff19c0c4d60 0x7ff19c0c51f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff194009d90 tx=0x7ff19402a5d0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.246+0000 7ff1a0c00640 1 -- 172.21.15.28:0/3964069847 shutdown_connections 2024-09-13T16:44:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.246+0000 7ff1a0c00640 1 --2- 172.21.15.28:0/3964069847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff19c0c4d60 0x7ff19c0c51f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.246+0000 7ff1a0c00640 1 --2- 172.21.15.28:0/3964069847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff19c1568e0 0x7ff19c0c2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.246+0000 7ff1a0c00640 1 -- 172.21.15.28:0/3964069847 >> 172.21.15.28:0/3964069847 conn(0x7ff19c066040 msgr2=0x7ff19c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.246+0000 7ff1a0c00640 1 -- 172.21.15.28:0/3964069847 shutdown_connections 2024-09-13T16:44:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.246+0000 7ff1a0c00640 1 -- 172.21.15.28:0/3964069847 wait complete. 2024-09-13T16:44:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.248+0000 7ff1a0c00640 1 Processor -- start 2024-09-13T16:44:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.248+0000 7ff1a0c00640 1 -- start start 2024-09-13T16:44:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.249+0000 7ff1a0c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff19c0c4d60 0x7ff19c0c02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.249+0000 7ff1a0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c150ca0 0x7ff19c157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.249+0000 7ff1a0c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff19c1568e0 0x7ff19c0bee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.249+0000 7ff1a0c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c0c0a20 con 0x7ff19c150ca0 2024-09-13T16:44:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.249+0000 7ff1a0c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c0c0d20 con 0x7ff19c1568e0 2024-09-13T16:44:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.249+0000 7ff1a0c00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c0c0fe0 con 0x7ff19c0c4d60 2024-09-13T16:44:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.249+0000 7ff191400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c150ca0 0x7ff19c157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.249+0000 7ff191400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c150ca0 0x7ff19c157cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53704/0 (socket says 172.21.15.28:53704) 2024-09-13T16:44:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.249+0000 7ff191400640 1 -- 172.21.15.28:0/3643782841 learned_addr learned my addr 172.21.15.28:0/3643782841 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.249+0000 7ff19a800640 1 --2- 172.21.15.28:0/3643782841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff19c1568e0 0x7ff19c0bee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.249+0000 7ff199e00640 1 --2- 172.21.15.28:0/3643782841 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff19c0c4d60 0x7ff19c0c02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.249+0000 7ff191400640 1 -- 172.21.15.28:0/3643782841 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff19c0c4d60 msgr2=0x7ff19c0c02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.249+0000 7ff191400640 1 --2- 172.21.15.28:0/3643782841 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff19c0c4d60 0x7ff19c0c02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.249+0000 7ff191400640 1 -- 172.21.15.28:0/3643782841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff19c1568e0 msgr2=0x7ff19c0bee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.249+0000 7ff191400640 1 --2- 172.21.15.28:0/3643782841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff19c1568e0 0x7ff19c0bee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.249+0000 7ff191400640 1 -- 172.21.15.28:0/3643782841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff194009a70 con 0x7ff19c150ca0 2024-09-13T16:44:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.249+0000 7ff19a800640 1 --2- 172.21.15.28:0/3643782841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff19c1568e0 0x7ff19c0bee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.250+0000 7ff191400640 1 --2- 172.21.15.28:0/3643782841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c150ca0 0x7ff19c157cd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff18400e9b0 tx=0x7ff18400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.250+0000 7ff192a00640 1 -- 172.21.15.28:0/3643782841 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff184009670 con 0x7ff19c150ca0 2024-09-13T16:44:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.250+0000 7ff192a00640 1 -- 172.21.15.28:0/3643782841 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18400c9a0 con 0x7ff19c150ca0 2024-09-13T16:44:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.250+0000 7ff192a00640 1 -- 172.21.15.28:0/3643782841 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18400cd50 con 0x7ff19c150ca0 2024-09-13T16:44:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.250+0000 7ff1a0c00640 1 -- 172.21.15.28:0/3643782841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19c14aa90 con 0x7ff19c150ca0 2024-09-13T16:44:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.250+0000 7ff1a0c00640 1 -- 172.21.15.28:0/3643782841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19c159c20 con 0x7ff19c150ca0 2024-09-13T16:44:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.251+0000 7ff1a0c00640 1 -- 172.21.15.28:0/3643782841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19c0c3110 con 0x7ff19c150ca0 2024-09-13T16:44:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.252+0000 7ff192a00640 1 -- 172.21.15.28:0/3643782841 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff184013470 con 0x7ff19c150ca0 2024-09-13T16:44:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.253+0000 7ff192a00640 1 --2- 172.21.15.28:0/3643782841 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff17806ac00 0x7ff17806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.253+0000 7ff192a00640 1 -- 172.21.15.28:0/3643782841 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff184089550 con 0x7ff19c150ca0 2024-09-13T16:44:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.253+0000 7ff199e00640 1 --2- 172.21.15.28:0/3643782841 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff17806ac00 0x7ff17806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.254+0000 7ff199e00640 1 --2- 172.21.15.28:0/3643782841 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff17806ac00 0x7ff17806d0c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff18c000f60 tx=0x7ff18c007d50 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.257+0000 7ff192a00640 1 -- 172.21.15.28:0/3643782841 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff1840547d0 con 0x7ff19c150ca0 2024-09-13T16:44:24.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.468+0000 7ff1a0c00640 1 -- 172.21.15.28:0/3643782841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff19c0c41b0 con 0x7ff19c150ca0 2024-09-13T16:44:24.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.469+0000 7ff192a00640 1 -- 172.21.15.28:0/3643782841 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6308 (secure 0 0 0) 0x7ff184057df0 con 0x7ff19c150ca0 2024-09-13T16:44:24.470 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:24.471 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4227,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/355570369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":355570369},{"type":"v1","addr":"172.21.15.28:6835","nonce":355570369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:43:54.969699+0000","modified":"2024-09-13T16:44:20.667541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4250,"mds_2":4244,"mds_3":4235,"mds_4":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"f","rank":4,"incarnation":16,"state":"up:creating","state_seq":1,"addr":"172.21.15.191:6839/2366383263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6837","nonce":2366383263},{"type":"v1","addr":"172.21.15.191:6839","nonce":2366383263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4235":{"gid":4235,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.191:6838/2702860767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2702860767},{"type":"v1","addr":"172.21.15.191:6838","nonce":2702860767}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/2240589617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2240589617},{"type":"v1","addr":"172.21.15.28:6833","nonce":2240589617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6837/4068563017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":4068563017},{"type":"v1","addr":"172.21.15.28:6837","nonce":4068563017}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4258":{"gid":4258,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/704943585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":704943585},{"type":"v1","addr":"172.21.15.191:6835","nonce":704943585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T16:44:24.471 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 16 2024-09-13T16:44:24.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.471+0000 7ff1a0c00640 1 -- 172.21.15.28:0/3643782841 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff17806ac00 msgr2=0x7ff17806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:24.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.471+0000 7ff1a0c00640 1 --2- 172.21.15.28:0/3643782841 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff17806ac00 0x7ff17806d0c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff18c000f60 tx=0x7ff18c007d50 comp rx=0 tx=0).stop 2024-09-13T16:44:24.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.471+0000 7ff1a0c00640 1 -- 172.21.15.28:0/3643782841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c150ca0 msgr2=0x7ff19c157cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:24.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.471+0000 7ff1a0c00640 1 --2- 172.21.15.28:0/3643782841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c150ca0 0x7ff19c157cd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff18400e9b0 tx=0x7ff18400ee70 comp rx=0 tx=0).stop 2024-09-13T16:44:24.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.471+0000 7ff1a0c00640 1 -- 172.21.15.28:0/3643782841 shutdown_connections 2024-09-13T16:44:24.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.471+0000 7ff1a0c00640 1 --2- 172.21.15.28:0/3643782841 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff17806ac00 0x7ff17806d0c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.471+0000 7ff1a0c00640 1 --2- 172.21.15.28:0/3643782841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff19c1568e0 0x7ff19c0bee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.471+0000 7ff1a0c00640 1 --2- 172.21.15.28:0/3643782841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c150ca0 0x7ff19c157cd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.471+0000 7ff1a0c00640 1 --2- 172.21.15.28:0/3643782841 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff19c0c4d60 0x7ff19c0c02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.471+0000 7ff1a0c00640 1 -- 172.21.15.28:0/3643782841 >> 172.21.15.28:0/3643782841 conn(0x7ff19c066040 msgr2=0x7ff19c0ca440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:24.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.471+0000 7ff1a0c00640 1 -- 172.21.15.28:0/3643782841 shutdown_connections 2024-09-13T16:44:24.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.471+0000 7ff1a0c00640 1 -- 172.21.15.28:0/3643782841 wait complete. 2024-09-13T16:44:24.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:44:24.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.641+0000 7f3245000640 1 Processor -- start 2024-09-13T16:44:24.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.641+0000 7f3245000640 1 -- start start 2024-09-13T16:44:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.642+0000 7f3245000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f32400c5bd0 0x7f3240156150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.643+0000 7f3245000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32400c3670 0x7f32400c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.643+0000 7f3245000640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f324014ca00 con 0x7f32400c1ea0 2024-09-13T16:44:24.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.643+0000 7f3245000640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32400c2dc0 con 0x7f32400c5bd0 2024-09-13T16:44:24.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.643+0000 7f3245000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3240155780 con 0x7f32400c3670 2024-09-13T16:44:24.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.643+0000 7f3235400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32400c3670 0x7f32400c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:24.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.643+0000 7f323de00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f32400c5bd0 0x7f3240156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:24.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.643+0000 7f3235400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32400c3670 0x7f32400c1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:60526/0 (socket says 172.21.15.28:60526) 2024-09-13T16:44:24.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.643+0000 7f3235400640 1 -- 172.21.15.28:0/1112477380 learned_addr learned my addr 172.21.15.28:0/1112477380 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.643+0000 7f3235400640 1 -- 172.21.15.28:0/1112477380 >> v2:172.21.15.191:3300/0 conn(0x7f32400c5bd0 msgr2=0x7f3240156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.643+0000 7f3235400640 1 --2- 172.21.15.28:0/1112477380 >> v2:172.21.15.191:3300/0 conn(0x7f32400c5bd0 0x7f3240156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.643+0000 7f3235400640 1 -- 172.21.15.28:0/1112477380 >> v1:172.21.15.191:6789/0 conn(0x7f32400c1ea0 legacy=0x7f32400c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.643+0000 7f3235400640 1 -- 172.21.15.28:0/1112477380 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3240155990 con 0x7f32400c3670 2024-09-13T16:44:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.644+0000 7f3235400640 1 --2- 172.21.15.28:0/1112477380 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32400c3670 0x7f32400c1960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3228009870 tx=0x7f322802a2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4110fd3c80807254 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.645+0000 7f323d400640 1 -- 172.21.15.28:0/1112477380 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228032640 con 0x7f32400c3670 2024-09-13T16:44:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.645+0000 7f323d400640 1 -- 172.21.15.28:0/1112477380 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32280327a0 con 0x7f32400c3670 2024-09-13T16:44:24.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.645+0000 7f323d400640 1 -- 172.21.15.28:0/1112477380 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228032b30 con 0x7f32400c3670 2024-09-13T16:44:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.645+0000 7f3245000640 1 -- 172.21.15.28:0/1112477380 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32400c3670 msgr2=0x7f32400c1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.645+0000 7f3245000640 1 --2- 172.21.15.28:0/1112477380 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32400c3670 0x7f32400c1960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3228009870 tx=0x7f322802a2e0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.645+0000 7f3245000640 1 -- 172.21.15.28:0/1112477380 shutdown_connections 2024-09-13T16:44:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.645+0000 7f3245000640 1 --2- 172.21.15.28:0/1112477380 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32400c3670 0x7f32400c1960 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.645+0000 7f3245000640 1 --2- 172.21.15.28:0/1112477380 >> v2:172.21.15.191:3300/0 conn(0x7f32400c5bd0 0x7f3240156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.645+0000 7f3245000640 1 -- 172.21.15.28:0/1112477380 >> 172.21.15.28:0/1112477380 conn(0x7f3240066040 msgr2=0x7f3240069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.645+0000 7f3245000640 1 -- 172.21.15.28:0/1112477380 shutdown_connections 2024-09-13T16:44:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.645+0000 7f3245000640 1 -- 172.21.15.28:0/1112477380 wait complete. 2024-09-13T16:44:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.647+0000 7f3245000640 1 Processor -- start 2024-09-13T16:44:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.647+0000 7f3245000640 1 -- start start 2024-09-13T16:44:24.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.648+0000 7f3245000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32400c1ea0 0x7f32401f1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:24.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.648+0000 7f3245000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32400c3670 0x7f3240158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:24.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.648+0000 7f3245000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32400c5bd0 0x7f3240159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:24.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.648+0000 7f3245000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3240156a80 con 0x7f32400c1ea0 2024-09-13T16:44:24.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.648+0000 7f3245000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3240150750 con 0x7f32400c5bd0 2024-09-13T16:44:24.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.648+0000 7f3245000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3240157eb0 con 0x7f32400c3670 2024-09-13T16:44:24.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.648+0000 7f323de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32400c1ea0 0x7f32401f1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:24.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.648+0000 7f3235400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32400c3670 0x7f3240158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:24.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.648+0000 7f3235400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32400c3670 0x7f3240158bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:60534/0 (socket says 172.21.15.28:60534) 2024-09-13T16:44:24.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.648+0000 7f323e800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32400c5bd0 0x7f3240159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:24.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.648+0000 7f3235400640 1 -- 172.21.15.28:0/774368696 learned_addr learned my addr 172.21.15.28:0/774368696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:24.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.648+0000 7f3235400640 1 -- 172.21.15.28:0/774368696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32400c5bd0 msgr2=0x7f3240159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:24.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.648+0000 7f3235400640 1 --2- 172.21.15.28:0/774368696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32400c5bd0 0x7f3240159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.648+0000 7f3235400640 1 -- 172.21.15.28:0/774368696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32400c1ea0 msgr2=0x7f32401f1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:24.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.648+0000 7f3235400640 1 --2- 172.21.15.28:0/774368696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32400c1ea0 0x7f32401f1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.648+0000 7f3235400640 1 -- 172.21.15.28:0/774368696 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32280094d0 con 0x7f32400c3670 2024-09-13T16:44:24.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.649+0000 7f3235400640 1 --2- 172.21.15.28:0/774368696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32400c3670 0x7f3240158bc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3228000c00 tx=0x7f32280046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.649+0000 7f3236a00640 1 -- 172.21.15.28:0/774368696 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322803d830 con 0x7f32400c3670 2024-09-13T16:44:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.649+0000 7f3236a00640 1 -- 172.21.15.28:0/774368696 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322803d990 con 0x7f32400c3670 2024-09-13T16:44:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.649+0000 7f3236a00640 1 -- 172.21.15.28:0/774368696 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322803dd40 con 0x7f32400c3670 2024-09-13T16:44:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.649+0000 7f3245000640 1 -- 172.21.15.28:0/774368696 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3240158730 con 0x7f32400c3670 2024-09-13T16:44:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.649+0000 7f3245000640 1 -- 172.21.15.28:0/774368696 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32401580e0 con 0x7f32400c3670 2024-09-13T16:44:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.650+0000 7f3245000640 1 -- 172.21.15.28:0/774368696 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3240157bb0 con 0x7f32400c3670 2024-09-13T16:44:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.651+0000 7f3236a00640 1 -- 172.21.15.28:0/774368696 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3228039040 con 0x7f32400c3670 2024-09-13T16:44:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.652+0000 7f3236a00640 1 --2- 172.21.15.28:0/774368696 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f321c06ac30 0x7f321c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.652+0000 7f3236a00640 1 -- 172.21.15.28:0/774368696 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32280a9cc0 con 0x7f32400c3670 2024-09-13T16:44:24.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.652+0000 7f323de00640 1 --2- 172.21.15.28:0/774368696 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f321c06ac30 0x7f321c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:24.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.653+0000 7f323de00640 1 --2- 172.21.15.28:0/774368696 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f321c06ac30 0x7f321c06d0f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3230009aa0 tx=0x7f3230009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.656+0000 7f3236a00640 1 -- 172.21.15.28:0/774368696 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f32280b0080 con 0x7f32400c3670 2024-09-13T16:44:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.837+0000 7f3245000640 1 -- 172.21.15.28:0/774368696 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3240070680 con 0x7f32400c3670 2024-09-13T16:44:24.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.839+0000 7f3236a00640 1 -- 172.21.15.28:0/774368696 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16481 (secure 0 0 0) 0x7f3228074f40 con 0x7f32400c3670 2024-09-13T16:44:24.841 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:24.841 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:43:55.971358+0000","last_up_change":"2024-09-13T16:43:49.476313+0000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:43:49.643606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:43:52.828510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:43:53.945874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6801","nonce":2964323435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6803","nonce":2964323435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6807","nonce":2964323435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6805","nonce":2964323435}]},"public_addr":"172.21.15.28:6801/2964323435","cluster_addr":"172.21.15.28:6803/2964323435","heartbeat_back_addr":"172.21.15.28:6807/2964323435","heartbeat_front_addr":"172.21.15.28:6805/2964323435","state":["exists","up"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6809","nonce":23768570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6811","nonce":23768570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6815","nonce":23768570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6813","nonce":23768570}]},"public_addr":"172.21.15.28:6809/23768570","cluster_addr":"172.21.15.28:6811/23768570","heartbeat_back_addr":"172.21.15.28:6815/23768570","heartbeat_front_addr":"172.21.15.28:6813/23768570","state":["exists","up"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6817","nonce":886887303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6819","nonce":886887303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6823","nonce":886887303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6821","nonce":886887303}]},"public_addr":"172.21.15.28:6817/886887303","cluster_addr":"172.21.15.28:6819/886887303","heartbeat_back_addr":"172.21.15.28:6823/886887303","heartbeat_front_addr":"172.21.15.28:6821/886887303","state":["exists","up"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6825","nonce":2616036674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6827","nonce":2616036674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6831","nonce":2616036674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6829","nonce":2616036674}]},"public_addr":"172.21.15.28:6825/2616036674","cluster_addr":"172.21.15.28:6827/2616036674","heartbeat_back_addr":"172.21.15.28:6831/2616036674","heartbeat_front_addr":"172.21.15.28:6829/2616036674","state":["exists","up"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6801","nonce":2817102450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6803","nonce":2817102450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6807","nonce":2817102450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6805","nonce":2817102450}]},"public_addr":"172.21.15.191:6801/2817102450","cluster_addr":"172.21.15.191:6803/2817102450","heartbeat_back_addr":"172.21.15.191:6807/2817102450","heartbeat_front_addr":"172.21.15.191:6805/2817102450","state":["exists","up"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6817","nonce":2503027971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6819","nonce":2503027971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6823","nonce":2503027971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6821","nonce":2503027971}]},"public_addr":"172.21.15.191:6817/2503027971","cluster_addr":"172.21.15.191:6819/2503027971","heartbeat_back_addr":"172.21.15.191:6823/2503027971","heartbeat_front_addr":"172.21.15.191:6821/2503027971","state":["exists","up"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6809","nonce":3006595088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6811","nonce":3006595088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6815","nonce":3006595088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6813","nonce":3006595088}]},"public_addr":"172.21.15.191:6809/3006595088","cluster_addr":"172.21.15.191:6811/3006595088","heartbeat_back_addr":"172.21.15.191:6815/3006595088","heartbeat_front_addr":"172.21.15.191:6813/3006595088","state":["exists","up"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6825","nonce":1477327720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6827","nonce":1477327720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6831","nonce":1477327720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6829","nonce":1477327720}]},"public_addr":"172.21.15.191:6825/1477327720","cluster_addr":"172.21.15.191:6827/1477327720","heartbeat_back_addr":"172.21.15.191:6831/1477327720","heartbeat_front_addr":"172.21.15.191:6829/1477327720","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.917250+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.954351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.017329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.046221+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.707467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.726947+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.782321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.764850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:44:24.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.841+0000 7f3245000640 1 -- 172.21.15.28:0/774368696 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f321c06ac30 msgr2=0x7f321c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:24.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.841+0000 7f3245000640 1 --2- 172.21.15.28:0/774368696 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f321c06ac30 0x7f321c06d0f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3230009aa0 tx=0x7f3230009190 comp rx=0 tx=0).stop 2024-09-13T16:44:24.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.841+0000 7f3245000640 1 -- 172.21.15.28:0/774368696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32400c3670 msgr2=0x7f3240158bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:24.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.841+0000 7f3245000640 1 --2- 172.21.15.28:0/774368696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32400c3670 0x7f3240158bc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3228000c00 tx=0x7f32280046c0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.841+0000 7f3245000640 1 -- 172.21.15.28:0/774368696 shutdown_connections 2024-09-13T16:44:24.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.841+0000 7f3245000640 1 --2- 172.21.15.28:0/774368696 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f321c06ac30 0x7f321c06d0f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.841+0000 7f3245000640 1 --2- 172.21.15.28:0/774368696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f32400c5bd0 0x7f3240159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.841+0000 7f3245000640 1 --2- 172.21.15.28:0/774368696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32400c3670 0x7f3240158bc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.841+0000 7f3245000640 1 --2- 172.21.15.28:0/774368696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32400c1ea0 0x7f32401f1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:24.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.841+0000 7f3245000640 1 -- 172.21.15.28:0/774368696 >> 172.21.15.28:0/774368696 conn(0x7f3240066040 msgr2=0x7f32400c6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:24.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.841+0000 7f3245000640 1 -- 172.21.15.28:0/774368696 shutdown_connections 2024-09-13T16:44:24.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:24.841+0000 7f3245000640 1 -- 172.21.15.28:0/774368696 wait complete. 2024-09-13T16:44:24.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:44:25.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.012+0000 7f09e9c00640 1 Processor -- start 2024-09-13T16:44:25.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.012+0000 7f09e9c00640 1 -- start start 2024-09-13T16:44:25.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.013+0000 7f09e9c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e4152e50 0x7f09e415c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:25.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.013+0000 7f09e9c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09e40c5420 0x7f09e40cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:25.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.013+0000 7f09e9c00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e40c59b0 con 0x7f09e41589e0 2024-09-13T16:44:25.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.013+0000 7f09e9c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e415cf20 con 0x7f09e40c5420 2024-09-13T16:44:25.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.013+0000 7f09e9c00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e406a400 con 0x7f09e4152e50 2024-09-13T16:44:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.013+0000 7f09e3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09e40c5420 0x7f09e40cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.013+0000 7f09e2a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e4152e50 0x7f09e415c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.013+0000 7f09e2a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e4152e50 0x7f09e415c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:60536/0 (socket says 172.21.15.28:60536) 2024-09-13T16:44:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.013+0000 7f09e2a00640 1 -- 172.21.15.28:0/3655346560 learned_addr learned my addr 172.21.15.28:0/3655346560 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.013+0000 7f09da000640 1 --1- 172.21.15.28:0/3655346560 >> v1:172.21.15.191:6789/0 conn(0x7f09e41589e0 0x7f09e40c6f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:47446/0 (socket says 172.21.15.28:47446) 2024-09-13T16:44:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.014+0000 7f09e2a00640 1 -- 172.21.15.28:0/3655346560 >> v2:172.21.15.28:3300/0 conn(0x7f09e40c5420 msgr2=0x7f09e40cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.014+0000 7f09e2a00640 1 --2- 172.21.15.28:0/3655346560 >> v2:172.21.15.28:3300/0 conn(0x7f09e40c5420 0x7f09e40cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.014+0000 7f09e2a00640 1 -- 172.21.15.28:0/3655346560 >> v1:172.21.15.191:6789/0 conn(0x7f09e41589e0 legacy=0x7f09e40c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.014+0000 7f09e2a00640 1 -- 172.21.15.28:0/3655346560 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e406a6e0 con 0x7f09e4152e50 2024-09-13T16:44:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.014+0000 7f09e3400640 1 --2- 172.21.15.28:0/3655346560 >> v2:172.21.15.28:3300/0 conn(0x7f09e40c5420 0x7f09e40cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.014+0000 7f09e2000640 1 -- 172.21.15.28:0/3655346560 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20746287 0 0) 0x7f09e40c59b0 con 0x7f09e41589e0 2024-09-13T16:44:25.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.015+0000 7f09e2a00640 1 --2- 172.21.15.28:0/3655346560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e4152e50 0x7f09e415c9e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009870 tx=0x7f09d402a2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18e3f4a058133314 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:25.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.015+0000 7f09e2000640 1 -- 172.21.15.28:0/3655346560 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d4032640 con 0x7f09e4152e50 2024-09-13T16:44:25.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.015+0000 7f09e2000640 1 -- 172.21.15.28:0/3655346560 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d40327a0 con 0x7f09e4152e50 2024-09-13T16:44:25.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.015+0000 7f09e2000640 1 -- 172.21.15.28:0/3655346560 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d4032b30 con 0x7f09e4152e50 2024-09-13T16:44:25.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.016+0000 7f09e9c00640 1 -- 172.21.15.28:0/3655346560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e4152e50 msgr2=0x7f09e415c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:25.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.016+0000 7f09e9c00640 1 --2- 172.21.15.28:0/3655346560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e4152e50 0x7f09e415c9e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009870 tx=0x7f09d402a2e0 comp rx=0 tx=0).stop 2024-09-13T16:44:25.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.016+0000 7f09e9c00640 1 -- 172.21.15.28:0/3655346560 shutdown_connections 2024-09-13T16:44:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.016+0000 7f09e9c00640 1 --2- 172.21.15.28:0/3655346560 >> v2:172.21.15.28:3300/0 conn(0x7f09e40c5420 0x7f09e40cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.016+0000 7f09e9c00640 1 --2- 172.21.15.28:0/3655346560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e4152e50 0x7f09e415c9e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.016+0000 7f09e9c00640 1 -- 172.21.15.28:0/3655346560 >> 172.21.15.28:0/3655346560 conn(0x7f09e406d530 msgr2=0x7f09e406af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.016+0000 7f09e9c00640 1 -- 172.21.15.28:0/3655346560 shutdown_connections 2024-09-13T16:44:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.016+0000 7f09e9c00640 1 -- 172.21.15.28:0/3655346560 wait complete. 2024-09-13T16:44:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.017+0000 7f09e9c00640 1 Processor -- start 2024-09-13T16:44:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.018+0000 7f09e9c00640 1 -- start start 2024-09-13T16:44:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.018+0000 7f09e9c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e40c5420 0x7f09e40c9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.018+0000 7f09e9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e4152e50 0x7f09e41f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.018+0000 7f09e9c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09e41589e0 0x7f09e4148f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.018+0000 7f09e9c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e4149c50 con 0x7f09e4152e50 2024-09-13T16:44:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.018+0000 7f09e9c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e40c9990 con 0x7f09e41589e0 2024-09-13T16:44:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.018+0000 7f09e9c00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e41f25b0 con 0x7f09e40c5420 2024-09-13T16:44:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.019+0000 7f09e2a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e40c5420 0x7f09e40c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.019+0000 7f09e2a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e40c5420 0x7f09e40c9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:60546/0 (socket says 172.21.15.28:60546) 2024-09-13T16:44:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.019+0000 7f09e2a00640 1 -- 172.21.15.28:0/4062282745 learned_addr learned my addr 172.21.15.28:0/4062282745 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.019+0000 7f09e3400640 1 --2- 172.21.15.28:0/4062282745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09e41589e0 0x7f09e4148f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.019+0000 7f09da000640 1 --2- 172.21.15.28:0/4062282745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e4152e50 0x7f09e41f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.019+0000 7f09e2a00640 1 -- 172.21.15.28:0/4062282745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09e41589e0 msgr2=0x7f09e4148f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.019+0000 7f09e2a00640 1 --2- 172.21.15.28:0/4062282745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09e41589e0 0x7f09e4148f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.019+0000 7f09e2a00640 1 -- 172.21.15.28:0/4062282745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e4152e50 msgr2=0x7f09e41f1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.019+0000 7f09e2a00640 1 --2- 172.21.15.28:0/4062282745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e4152e50 0x7f09e41f1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.019+0000 7f09e2a00640 1 -- 172.21.15.28:0/4062282745 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09dc009a70 con 0x7f09e40c5420 2024-09-13T16:44:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.019+0000 7f09e2a00640 1 --2- 172.21.15.28:0/4062282745 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e40c5420 0x7f09e40c9450 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009870 tx=0x7f09d40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.020+0000 7f09db400640 1 -- 172.21.15.28:0/4062282745 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d403d830 con 0x7f09e40c5420 2024-09-13T16:44:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.020+0000 7f09db400640 1 -- 172.21.15.28:0/4062282745 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d403d9b0 con 0x7f09e40c5420 2024-09-13T16:44:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.020+0000 7f09db400640 1 -- 172.21.15.28:0/4062282745 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d403dd60 con 0x7f09e40c5420 2024-09-13T16:44:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.020+0000 7f09e9c00640 1 -- 172.21.15.28:0/4062282745 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d40094d0 con 0x7f09e40c5420 2024-09-13T16:44:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.020+0000 7f09e9c00640 1 -- 172.21.15.28:0/4062282745 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e414f3b0 con 0x7f09e40c5420 2024-09-13T16:44:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.021+0000 7f09e9c00640 1 -- 172.21.15.28:0/4062282745 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e40c59b0 con 0x7f09e40c5420 2024-09-13T16:44:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.022+0000 7f09db400640 1 -- 172.21.15.28:0/4062282745 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f09d4039040 con 0x7f09e40c5420 2024-09-13T16:44:25.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.023+0000 7f09db400640 1 --2- 172.21.15.28:0/4062282745 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f09c006ac00 0x7f09c006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:25.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.023+0000 7f09db400640 1 -- 172.21.15.28:0/4062282745 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09d40a9e30 con 0x7f09e40c5420 2024-09-13T16:44:25.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.023+0000 7f09da000640 1 --2- 172.21.15.28:0/4062282745 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f09c006ac00 0x7f09c006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:25.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.024+0000 7f09da000640 1 --2- 172.21.15.28:0/4062282745 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f09c006ac00 0x7f09c006d0c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f09e40cb4f0 tx=0x7f09cc007de0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:25.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.027+0000 7f09db400640 1 -- 172.21.15.28:0/4062282745 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f09d4074f90 con 0x7f09e40c5420 2024-09-13T16:44:25.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.241+0000 7f09e9c00640 1 -- 172.21.15.28:0/4062282745 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09e40cbf70 con 0x7f09e40c5420 2024-09-13T16:44:25.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.242+0000 7f09db400640 1 -- 172.21.15.28:0/4062282745 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6308 (secure 0 0 0) 0x7f09d4037090 con 0x7f09e40c5420 2024-09-13T16:44:25.243 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:25.243 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4227,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/355570369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":355570369},{"type":"v1","addr":"172.21.15.28:6835","nonce":355570369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:43:54.969699+0000","modified":"2024-09-13T16:44:20.667541+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4250,"mds_2":4244,"mds_3":4235,"mds_4":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"f","rank":4,"incarnation":16,"state":"up:creating","state_seq":1,"addr":"172.21.15.191:6839/2366383263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6837","nonce":2366383263},{"type":"v1","addr":"172.21.15.191:6839","nonce":2366383263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4235":{"gid":4235,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.191:6838/2702860767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2702860767},{"type":"v1","addr":"172.21.15.191:6838","nonce":2702860767}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/2240589617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2240589617},{"type":"v1","addr":"172.21.15.28:6833","nonce":2240589617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6837/4068563017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":4068563017},{"type":"v1","addr":"172.21.15.28:6837","nonce":4068563017}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4258":{"gid":4258,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/704943585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":704943585},{"type":"v1","addr":"172.21.15.191:6835","nonce":704943585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T16:44:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 16 2024-09-13T16:44:25.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.244+0000 7f09e9c00640 1 -- 172.21.15.28:0/4062282745 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f09c006ac00 msgr2=0x7f09c006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.244+0000 7f09e9c00640 1 --2- 172.21.15.28:0/4062282745 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f09c006ac00 0x7f09c006d0c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f09e40cb4f0 tx=0x7f09cc007de0 comp rx=0 tx=0).stop 2024-09-13T16:44:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.244+0000 7f09e9c00640 1 -- 172.21.15.28:0/4062282745 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e40c5420 msgr2=0x7f09e40c9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.244+0000 7f09e9c00640 1 --2- 172.21.15.28:0/4062282745 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e40c5420 0x7f09e40c9450 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009870 tx=0x7f09d40046c0 comp rx=0 tx=0).stop 2024-09-13T16:44:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.244+0000 7f09e9c00640 1 -- 172.21.15.28:0/4062282745 shutdown_connections 2024-09-13T16:44:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.244+0000 7f09e9c00640 1 --2- 172.21.15.28:0/4062282745 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f09c006ac00 0x7f09c006d0c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.244+0000 7f09e9c00640 1 --2- 172.21.15.28:0/4062282745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09e41589e0 0x7f09e4148f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.244+0000 7f09e9c00640 1 --2- 172.21.15.28:0/4062282745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e4152e50 0x7f09e41f1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.244+0000 7f09e9c00640 1 --2- 172.21.15.28:0/4062282745 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f09e40c5420 0x7f09e40c9450 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.244+0000 7f09e9c00640 1 -- 172.21.15.28:0/4062282745 >> 172.21.15.28:0/4062282745 conn(0x7f09e406d530 msgr2=0x7f09e4158df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.244+0000 7f09e9c00640 1 -- 172.21.15.28:0/4062282745 shutdown_connections 2024-09-13T16:44:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:25.244+0000 7f09e9c00640 1 -- 172.21.15.28:0/4062282745 wait complete. 2024-09-13T16:44:25.865 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:43:54.969699+0000', 'modified': '2024-09-13T16:44:20.667541+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4250, 'mds_2': 4244, 'mds_3': 4235, 'mds_4': 4232}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4232': {'gid': 4232, 'name': 'f', 'rank': 4, 'incarnation': 16, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.191:6839/2366383263', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6837', 'nonce': 2366383263}, {'type': 'v1', 'addr': '172.21.15.191:6839', 'nonce': 2366383263}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4235': {'gid': 4235, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.191:6838/2702860767', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6836', 'nonce': 2702860767}, {'type': 'v1', 'addr': '172.21.15.191:6838', 'nonce': 2702860767}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4244': {'gid': 4244, 'name': 'c', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.28:6833/2240589617', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2240589617}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 2240589617}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4250': {'gid': 4250, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.28:6837/4068563017', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6836', 'nonce': 4068563017}, {'type': 'v1', 'addr': '172.21.15.28:6837', 'nonce': 4068563017}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4258': {'gid': 4258, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.191:6835/704943585', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 704943585}, {'type': 'v1', 'addr': '172.21.15.191:6835', 'nonce': 704943585}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:44:25.866 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4232:up:creating 2024-09-13T16:44:26.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:44:27.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.032+0000 7fa99ee00640 1 Processor -- start 2024-09-13T16:44:27.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.032+0000 7fa99ee00640 1 -- start start 2024-09-13T16:44:27.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.033+0000 7fa99ee00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa99806b5e0 0x7fa998159760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:27.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.033+0000 7fa99ee00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa998069e80 0x7fa998071760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:27.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.033+0000 7fa99ee00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9981592d0 con 0x7fa99806ad40 2024-09-13T16:44:27.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.033+0000 7fa99ee00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa998070680 con 0x7fa99806b5e0 2024-09-13T16:44:27.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.033+0000 7fa99ee00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa99806b120 con 0x7fa998069e80 2024-09-13T16:44:27.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.034+0000 7fa997e00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fa99806ad40 0x7fa998070b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:47462/0 (socket says 172.21.15.28:47462) 2024-09-13T16:44:27.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.034+0000 7fa997e00640 1 -- 172.21.15.28:0/2058846308 learned_addr learned my addr 172.21.15.28:0/2058846308 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:27.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.034+0000 7fa997400640 1 --2- 172.21.15.28:0/2058846308 >> v2:172.21.15.28:3300/0 conn(0x7fa99806b5e0 0x7fa998159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:27.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.034+0000 7fa99ce00640 1 --2- 172.21.15.28:0/2058846308 >> v2:172.21.15.191:3300/0 conn(0x7fa998069e80 0x7fa998071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:27.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.034+0000 7fa996a00640 1 -- 172.21.15.28:0/2058846308 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178534366 0 0) 0x7fa9981592d0 con 0x7fa99806ad40 2024-09-13T16:44:27.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.034+0000 7fa996a00640 1 -- 172.21.15.28:0/2058846308 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa97c003520 con 0x7fa99806ad40 2024-09-13T16:44:27.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.034+0000 7fa997400640 1 -- 172.21.15.28:0/2058846308 >> v2:172.21.15.191:3300/0 conn(0x7fa998069e80 msgr2=0x7fa998071760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:27.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.034+0000 7fa997400640 1 --2- 172.21.15.28:0/2058846308 >> v2:172.21.15.191:3300/0 conn(0x7fa998069e80 0x7fa998071760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:27.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.034+0000 7fa997400640 1 -- 172.21.15.28:0/2058846308 >> v1:172.21.15.191:6789/0 conn(0x7fa99806ad40 legacy=0x7fa998070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:27.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.034+0000 7fa997400640 1 -- 172.21.15.28:0/2058846308 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99806a2b0 con 0x7fa99806b5e0 2024-09-13T16:44:27.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.035+0000 7fa997400640 1 --2- 172.21.15.28:0/2058846308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa99806b5e0 0x7fa998159760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa99806da30 tx=0x7fa98402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f2632cb8d10612 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:27.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.035+0000 7fa996a00640 1 -- 172.21.15.28:0/2058846308 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa984032440 con 0x7fa99806b5e0 2024-09-13T16:44:27.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.035+0000 7fa996a00640 1 -- 172.21.15.28:0/2058846308 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9840325a0 con 0x7fa99806b5e0 2024-09-13T16:44:27.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.035+0000 7fa996a00640 1 -- 172.21.15.28:0/2058846308 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa984032910 con 0x7fa99806b5e0 2024-09-13T16:44:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.036+0000 7fa99ee00640 1 -- 172.21.15.28:0/2058846308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa99806b5e0 msgr2=0x7fa998159760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.036+0000 7fa99ee00640 1 --2- 172.21.15.28:0/2058846308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa99806b5e0 0x7fa998159760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa99806da30 tx=0x7fa98402a250 comp rx=0 tx=0).stop 2024-09-13T16:44:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.036+0000 7fa99ee00640 1 -- 172.21.15.28:0/2058846308 shutdown_connections 2024-09-13T16:44:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.036+0000 7fa99ee00640 1 --2- 172.21.15.28:0/2058846308 >> v2:172.21.15.191:3300/0 conn(0x7fa998069e80 0x7fa998071760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.036+0000 7fa99ee00640 1 --2- 172.21.15.28:0/2058846308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa99806b5e0 0x7fa998159760 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.036+0000 7fa99ee00640 1 -- 172.21.15.28:0/2058846308 >> 172.21.15.28:0/2058846308 conn(0x7fa9980c7ad0 msgr2=0x7fa9980c7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.036+0000 7fa99ee00640 1 -- 172.21.15.28:0/2058846308 shutdown_connections 2024-09-13T16:44:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.036+0000 7fa99ee00640 1 -- 172.21.15.28:0/2058846308 wait complete. 2024-09-13T16:44:27.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.038+0000 7fa99ee00640 1 Processor -- start 2024-09-13T16:44:27.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.038+0000 7fa99ee00640 1 -- start start 2024-09-13T16:44:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.039+0000 7fa99ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa998069e80 0x7fa998151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.039+0000 7fa99ee00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa99806ad40 0x7fa9981ef480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.039+0000 7fa99ee00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa99806b5e0 0x7fa99814e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.039+0000 7fa99ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa998150f90 con 0x7fa998069e80 2024-09-13T16:44:27.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.039+0000 7fa99ee00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9981ef9c0 con 0x7fa99806b5e0 2024-09-13T16:44:27.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.039+0000 7fa99ee00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9980ad0a0 con 0x7fa99806ad40 2024-09-13T16:44:27.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.039+0000 7fa997e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa998069e80 0x7fa998151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:27.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.039+0000 7fa997e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa998069e80 0x7fa998151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53762/0 (socket says 172.21.15.28:53762) 2024-09-13T16:44:27.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.039+0000 7fa997e00640 1 -- 172.21.15.28:0/752352420 learned_addr learned my addr 172.21.15.28:0/752352420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:27.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.039+0000 7fa99ce00640 1 --2- 172.21.15.28:0/752352420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa99806b5e0 0x7fa99814e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:27.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.039+0000 7fa997400640 1 --2- 172.21.15.28:0/752352420 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa99806ad40 0x7fa9981ef480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:27.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.039+0000 7fa997e00640 1 -- 172.21.15.28:0/752352420 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa99806ad40 msgr2=0x7fa9981ef480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:27.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.039+0000 7fa997e00640 1 --2- 172.21.15.28:0/752352420 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa99806ad40 0x7fa9981ef480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:27.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.039+0000 7fa997e00640 1 -- 172.21.15.28:0/752352420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa99806b5e0 msgr2=0x7fa99814e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:27.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.039+0000 7fa997e00640 1 --2- 172.21.15.28:0/752352420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa99806b5e0 0x7fa99814e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:27.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.039+0000 7fa997e00640 1 -- 172.21.15.28:0/752352420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa984009400 con 0x7fa998069e80 2024-09-13T16:44:27.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.039+0000 7fa997e00640 1 --2- 172.21.15.28:0/752352420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa998069e80 0x7fa998151770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa98800c410 tx=0x7fa98800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:27.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.040+0000 7fa98fe00640 1 -- 172.21.15.28:0/752352420 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988005ce0 con 0x7fa998069e80 2024-09-13T16:44:27.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.040+0000 7fa98fe00640 1 -- 172.21.15.28:0/752352420 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa988005e40 con 0x7fa998069e80 2024-09-13T16:44:27.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.040+0000 7fa98fe00640 1 -- 172.21.15.28:0/752352420 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9880072f0 con 0x7fa998069e80 2024-09-13T16:44:27.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.040+0000 7fa99ee00640 1 -- 172.21.15.28:0/752352420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa998151180 con 0x7fa998069e80 2024-09-13T16:44:27.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.040+0000 7fa99ee00640 1 -- 172.21.15.28:0/752352420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa998152290 con 0x7fa998069e80 2024-09-13T16:44:27.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.041+0000 7fa99ee00640 1 -- 172.21.15.28:0/752352420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa998159760 con 0x7fa998069e80 2024-09-13T16:44:27.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.042+0000 7fa98fe00640 1 -- 172.21.15.28:0/752352420 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa988007450 con 0x7fa998069e80 2024-09-13T16:44:27.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.043+0000 7fa98fe00640 1 --2- 172.21.15.28:0/752352420 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa97806ac30 0x7fa97806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:27.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.043+0000 7fa98fe00640 1 -- 172.21.15.28:0/752352420 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9880889b0 con 0x7fa998069e80 2024-09-13T16:44:27.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.043+0000 7fa997400640 1 --2- 172.21.15.28:0/752352420 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa97806ac30 0x7fa97806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:27.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.044+0000 7fa997400640 1 --2- 172.21.15.28:0/752352420 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa97806ac30 0x7fa97806d0f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa984034000 tx=0x7fa98402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:27.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.047+0000 7fa98fe00640 1 -- 172.21.15.28:0/752352420 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa9880576f0 con 0x7fa998069e80 2024-09-13T16:44:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.259+0000 7fa99ee00640 1 -- 172.21.15.28:0/752352420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9980c13e0 con 0x7fa998069e80 2024-09-13T16:44:27.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.260+0000 7fa98fe00640 1 -- 172.21.15.28:0/752352420 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6307 (secure 0 0 0) 0x7fa988017090 con 0x7fa998069e80 2024-09-13T16:44:27.261 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:27.261 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4227,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/355570369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":355570369},{"type":"v1","addr":"172.21.15.28:6835","nonce":355570369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:43:54.969699+0000","modified":"2024-09-13T16:44:26.685528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4250,"mds_2":4244,"mds_3":4235,"mds_4":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"f","rank":4,"incarnation":16,"state":"up:active","state_seq":10,"addr":"172.21.15.191:6839/2366383263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6837","nonce":2366383263},{"type":"v1","addr":"172.21.15.191:6839","nonce":2366383263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4235":{"gid":4235,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.191:6838/2702860767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2702860767},{"type":"v1","addr":"172.21.15.191:6838","nonce":2702860767}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/2240589617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2240589617},{"type":"v1","addr":"172.21.15.28:6833","nonce":2240589617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6837/4068563017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":4068563017},{"type":"v1","addr":"172.21.15.28:6837","nonce":4068563017}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4258":{"gid":4258,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/704943585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":704943585},{"type":"v1","addr":"172.21.15.191:6835","nonce":704943585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T16:44:27.262 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-09-13T16:44:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.262+0000 7fa99ee00640 1 -- 172.21.15.28:0/752352420 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa97806ac30 msgr2=0x7fa97806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.262+0000 7fa99ee00640 1 --2- 172.21.15.28:0/752352420 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa97806ac30 0x7fa97806d0f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa984034000 tx=0x7fa98402e000 comp rx=0 tx=0).stop 2024-09-13T16:44:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.262+0000 7fa99ee00640 1 -- 172.21.15.28:0/752352420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa998069e80 msgr2=0x7fa998151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.262+0000 7fa99ee00640 1 --2- 172.21.15.28:0/752352420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa998069e80 0x7fa998151770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa98800c410 tx=0x7fa98800c8d0 comp rx=0 tx=0).stop 2024-09-13T16:44:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.262+0000 7fa99ee00640 1 -- 172.21.15.28:0/752352420 shutdown_connections 2024-09-13T16:44:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.262+0000 7fa99ee00640 1 --2- 172.21.15.28:0/752352420 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa97806ac30 0x7fa97806d0f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.262+0000 7fa99ee00640 1 --2- 172.21.15.28:0/752352420 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa99806b5e0 0x7fa99814e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.262+0000 7fa99ee00640 1 --2- 172.21.15.28:0/752352420 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa99806ad40 0x7fa9981ef480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.262+0000 7fa99ee00640 1 --2- 172.21.15.28:0/752352420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa998069e80 0x7fa998151770 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.262+0000 7fa99ee00640 1 -- 172.21.15.28:0/752352420 >> 172.21.15.28:0/752352420 conn(0x7fa9980c7ad0 msgr2=0x7fa9980afc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.262+0000 7fa99ee00640 1 -- 172.21.15.28:0/752352420 shutdown_connections 2024-09-13T16:44:27.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:27.262+0000 7fa99ee00640 1 -- 172.21.15.28:0/752352420 wait complete. 2024-09-13T16:44:27.882 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:43:54.969699+0000', 'modified': '2024-09-13T16:44:26.685528+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4250, 'mds_2': 4244, 'mds_3': 4235, 'mds_4': 4232}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4232': {'gid': 4232, 'name': 'f', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.191:6839/2366383263', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6837', 'nonce': 2366383263}, {'type': 'v1', 'addr': '172.21.15.191:6839', 'nonce': 2366383263}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4235': {'gid': 4235, 'name': 'b', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.191:6838/2702860767', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6836', 'nonce': 2702860767}, {'type': 'v1', 'addr': '172.21.15.191:6838', 'nonce': 2702860767}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4244': {'gid': 4244, 'name': 'c', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.28:6833/2240589617', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2240589617}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 2240589617}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4250': {'gid': 4250, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.28:6837/4068563017', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6836', 'nonce': 4068563017}, {'type': 'v1', 'addr': '172.21.15.28:6837', 'nonce': 4068563017}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4258': {'gid': 4258, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.191:6835/704943585', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 704943585}, {'type': 'v1', 'addr': '172.21.15.191:6835', 'nonce': 704943585}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:44:27.883 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-13T16:44:27.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-09-13T16:44:28.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.046+0000 7f76a1a00640 1 Processor -- start 2024-09-13T16:44:28.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.046+0000 7f76a1a00640 1 -- start start 2024-09-13T16:44:28.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.047+0000 7f76a1a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f769c0c5bd0 0x7f769c156150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:28.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.048+0000 7f76a1a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f769c0c6ef0 0x7f769c0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:28.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.048+0000 7f76a1a00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f769c14ca00 con 0x7f769c0c3670 2024-09-13T16:44:28.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.048+0000 7f76a1a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f769c0c2dc0 con 0x7f769c0c5bd0 2024-09-13T16:44:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.048+0000 7f76a1a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f769c155780 con 0x7f769c0c6ef0 2024-09-13T16:44:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.048+0000 7f769aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f769c0c5bd0 0x7f769c156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.048+0000 7f7692000640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f769c0c3670 0x7f769c0c1960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:47474/0 (socket says 172.21.15.28:47474) 2024-09-13T16:44:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.048+0000 7f769b400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f769c0c6ef0 0x7f769c0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.048+0000 7f7692000640 1 -- 172.21.15.28:0/2758796723 learned_addr learned my addr 172.21.15.28:0/2758796723 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.048+0000 7f769aa00640 1 -- 172.21.15.28:0/2758796723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f769c0c6ef0 msgr2=0x7f769c0c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.048+0000 7f769aa00640 1 --2- 172.21.15.28:0/2758796723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f769c0c6ef0 0x7f769c0c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.048+0000 7f769aa00640 1 -- 172.21.15.28:0/2758796723 >> v1:172.21.15.191:6789/0 conn(0x7f769c0c3670 legacy=0x7f769c0c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:28.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.048+0000 7f769aa00640 1 -- 172.21.15.28:0/2758796723 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f769c155990 con 0x7f769c0c5bd0 2024-09-13T16:44:28.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.049+0000 7f769aa00640 1 --2- 172.21.15.28:0/2758796723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f769c0c5bd0 0x7f769c156150 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f768c0097a0 tx=0x7f768c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13efa949e6e8237e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:28.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.050+0000 7f769a000640 1 -- 172.21.15.28:0/2758796723 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c032440 con 0x7f769c0c5bd0 2024-09-13T16:44:28.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.050+0000 7f769a000640 1 -- 172.21.15.28:0/2758796723 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768c0325a0 con 0x7f769c0c5bd0 2024-09-13T16:44:28.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.050+0000 7f769a000640 1 -- 172.21.15.28:0/2758796723 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c032930 con 0x7f769c0c5bd0 2024-09-13T16:44:28.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.050+0000 7f76a1a00640 1 -- 172.21.15.28:0/2758796723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f769c0c5bd0 msgr2=0x7f769c156150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:28.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.050+0000 7f76a1a00640 1 --2- 172.21.15.28:0/2758796723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f769c0c5bd0 0x7f769c156150 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f768c0097a0 tx=0x7f768c02a210 comp rx=0 tx=0).stop 2024-09-13T16:44:28.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.050+0000 7f76a1a00640 1 -- 172.21.15.28:0/2758796723 shutdown_connections 2024-09-13T16:44:28.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.050+0000 7f76a1a00640 1 --2- 172.21.15.28:0/2758796723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f769c0c6ef0 0x7f769c0c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:28.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.050+0000 7f76a1a00640 1 --2- 172.21.15.28:0/2758796723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f769c0c5bd0 0x7f769c156150 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:28.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.050+0000 7f76a1a00640 1 -- 172.21.15.28:0/2758796723 >> 172.21.15.28:0/2758796723 conn(0x7f769c066040 msgr2=0x7f769c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.050+0000 7f76a1a00640 1 -- 172.21.15.28:0/2758796723 shutdown_connections 2024-09-13T16:44:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.050+0000 7f76a1a00640 1 -- 172.21.15.28:0/2758796723 wait complete. 2024-09-13T16:44:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.052+0000 7f76a1a00640 1 Processor -- start 2024-09-13T16:44:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.052+0000 7f76a1a00640 1 -- start start 2024-09-13T16:44:28.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.053+0000 7f76a1a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f769c0c3670 0x7f769c14c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:28.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.053+0000 7f76a1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f769c0c5bd0 0x7f769c1617b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:28.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.053+0000 7f76a1a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f769c0c6ef0 0x7f769c161cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:28.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.053+0000 7f76a1a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c145ab0 con 0x7f769c0c5bd0 2024-09-13T16:44:28.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.053+0000 7f76a1a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c153310 con 0x7f769c0c3670 2024-09-13T16:44:28.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.053+0000 7f76a1a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f769c1f2f90 con 0x7f769c0c6ef0 2024-09-13T16:44:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.053+0000 7f7692000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f769c0c5bd0 0x7f769c1617b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.053+0000 7f7692000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f769c0c5bd0 0x7f769c1617b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53786/0 (socket says 172.21.15.28:53786) 2024-09-13T16:44:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.053+0000 7f7692000640 1 -- 172.21.15.28:0/947809723 learned_addr learned my addr 172.21.15.28:0/947809723 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.053+0000 7f769b400640 1 --2- 172.21.15.28:0/947809723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f769c0c6ef0 0x7f769c161cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.053+0000 7f769aa00640 1 --2- 172.21.15.28:0/947809723 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f769c0c3670 0x7f769c14c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.054+0000 7f7692000640 1 -- 172.21.15.28:0/947809723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f769c0c6ef0 msgr2=0x7f769c161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.054+0000 7f7692000640 1 --2- 172.21.15.28:0/947809723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f769c0c6ef0 0x7f769c161cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.054+0000 7f7692000640 1 -- 172.21.15.28:0/947809723 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f769c0c3670 msgr2=0x7f769c14c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.054+0000 7f7692000640 1 --2- 172.21.15.28:0/947809723 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f769c0c3670 0x7f769c14c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.054+0000 7f7692000640 1 -- 172.21.15.28:0/947809723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c009400 con 0x7f769c0c5bd0 2024-09-13T16:44:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.054+0000 7f7692000640 1 --2- 172.21.15.28:0/947809723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f769c0c5bd0 0x7f769c1617b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f768400c530 tx=0x7f768400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.054+0000 7f7693400640 1 -- 172.21.15.28:0/947809723 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7684005ce0 con 0x7f769c0c5bd0 2024-09-13T16:44:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.054+0000 7f7693400640 1 -- 172.21.15.28:0/947809723 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7684005e40 con 0x7f769c0c5bd0 2024-09-13T16:44:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.054+0000 7f7693400640 1 -- 172.21.15.28:0/947809723 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7684007270 con 0x7f769c0c5bd0 2024-09-13T16:44:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.054+0000 7f76a1a00640 1 -- 172.21.15.28:0/947809723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f769c153560 con 0x7f769c0c5bd0 2024-09-13T16:44:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.054+0000 7f76a1a00640 1 -- 172.21.15.28:0/947809723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f769c161290 con 0x7f769c0c5bd0 2024-09-13T16:44:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.056+0000 7f7693400640 1 -- 172.21.15.28:0/947809723 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f76840073d0 con 0x7f769c0c5bd0 2024-09-13T16:44:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.057+0000 7f7693400640 1 --2- 172.21.15.28:0/947809723 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f767806ac00 0x7f767806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.057+0000 7f7693400640 1 -- 172.21.15.28:0/947809723 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7684088870 con 0x7f769c0c5bd0 2024-09-13T16:44:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.057+0000 7f769aa00640 1 --2- 172.21.15.28:0/947809723 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f767806ac00 0x7f767806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.058+0000 7f76a47fc740 1 Processor -- start 2024-09-13T16:44:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.058+0000 7f76a47fc740 1 -- start start 2024-09-13T16:44:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.058+0000 7f769aa00640 1 --2- 172.21.15.28:0/947809723 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f767806ac00 0x7f767806d0c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f768c034000 tx=0x7f768c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.059+0000 7f76a47fc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce72ad0 0x55d0fceaad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.059+0000 7f76a47fc740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce70300 0x55d0fce6ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.059+0000 7f76a47fc740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fceab2d0 0x55d0fce6f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.059+0000 7f76a47fc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fcddaab0 con 0x55d0fce72ad0 2024-09-13T16:44:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.059+0000 7f76a47fc740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fcdd8240 con 0x55d0fce70300 2024-09-13T16:44:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.059+0000 7f76a47fc740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d0fcdd3b30 con 0x55d0fceab2d0 2024-09-13T16:44:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.059+0000 7f769aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce72ad0 0x55d0fceaad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.059+0000 7f769aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce72ad0 0x55d0fceaad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53794/0 (socket says 172.21.15.28:53794) 2024-09-13T16:44:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.059+0000 7f769aa00640 1 -- 172.21.15.28:0/1462196494 learned_addr learned my addr 172.21.15.28:0/1462196494 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.059+0000 7f769b400640 1 --2- 172.21.15.28:0/1462196494 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fceab2d0 0x55d0fce6f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.059+0000 7f7692000640 1 --2- 172.21.15.28:0/1462196494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce70300 0x55d0fce6ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.060+0000 7f7692000640 1 -- 172.21.15.28:0/1462196494 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fceab2d0 msgr2=0x55d0fce6f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.060+0000 7f7692000640 1 --2- 172.21.15.28:0/1462196494 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fceab2d0 0x55d0fce6f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.060+0000 7f7692000640 1 -- 172.21.15.28:0/1462196494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce72ad0 msgr2=0x55d0fceaad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.060+0000 7f7692000640 1 --2- 172.21.15.28:0/1462196494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce72ad0 0x55d0fceaad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.060+0000 7f7692000640 1 -- 172.21.15.28:0/1462196494 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0fcd9ebd0 con 0x55d0fce70300 2024-09-13T16:44:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.060+0000 7f769aa00640 1 --2- 172.21.15.28:0/1462196494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce72ad0 0x55d0fceaad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.060+0000 7f7692000640 1 --2- 172.21.15.28:0/1462196494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce70300 0x55d0fce6ede0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7684086cd0 tx=0x7f7684057390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=180d3fa5bb67775 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.060+0000 7f768b400640 1 -- 172.21.15.28:0/1462196494 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7684007680 con 0x55d0fce70300 2024-09-13T16:44:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.060+0000 7f768b400640 1 -- 172.21.15.28:0/1462196494 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7684055ac0 con 0x55d0fce70300 2024-09-13T16:44:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.060+0000 7f768b400640 1 -- 172.21.15.28:0/1462196494 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7684053e30 con 0x55d0fce70300 2024-09-13T16:44:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.061+0000 7f76a47fc740 1 -- 172.21.15.28:0/1462196494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce70300 msgr2=0x55d0fce6ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.061+0000 7f76a47fc740 1 --2- 172.21.15.28:0/1462196494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce70300 0x55d0fce6ede0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7684086cd0 tx=0x7f7684057390 comp rx=0 tx=0).stop 2024-09-13T16:44:28.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.061+0000 7f76a47fc740 1 -- 172.21.15.28:0/1462196494 shutdown_connections 2024-09-13T16:44:28.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.061+0000 7f76a47fc740 1 --2- 172.21.15.28:0/1462196494 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fceab2d0 0x55d0fce6f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:28.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.061+0000 7f76a47fc740 1 --2- 172.21.15.28:0/1462196494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce70300 0x55d0fce6ede0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:28.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.061+0000 7f76a47fc740 1 --2- 172.21.15.28:0/1462196494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce72ad0 0x55d0fceaad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:28.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.061+0000 7f76a47fc740 1 -- 172.21.15.28:0/1462196494 >> 172.21.15.28:0/1462196494 conn(0x55d0fce711f0 msgr2=0x55d0fce715f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:28.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.061+0000 7f76a47fc740 1 -- 172.21.15.28:0/1462196494 shutdown_connections 2024-09-13T16:44:28.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.061+0000 7f76a47fc740 1 -- 172.21.15.28:0/1462196494 wait complete. 2024-09-13T16:44:28.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.061+0000 7f76a47fc740 1 -- start start 2024-09-13T16:44:28.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.062+0000 7f76a47fc740 1 Processor -- start 2024-09-13T16:44:28.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f76a47fc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce70300 0x55d0fcea05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:28.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f76a47fc740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce72ad0 0x55d0fce9c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:28.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f76a47fc740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fceab2d0 0x55d0fcea2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:28.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f76a47fc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fcd75260 con 0x55d0fce70300 2024-09-13T16:44:28.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f76a47fc740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fcd74f00 con 0x55d0fce72ad0 2024-09-13T16:44:28.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f76a47fc740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d0fcb3af20 con 0x55d0fceab2d0 2024-09-13T16:44:28.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f76a47fc740 10 client.0 _ll_register_callbacks cb 0x55d0fcdcd1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:44:28.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f7692000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce72ad0 0x55d0fce9c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:28.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f769b400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fceab2d0 0x55d0fcea2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:28.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f769aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce70300 0x55d0fcea05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:28.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f769b400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fceab2d0 0x55d0fcea2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:60602/0 (socket says 172.21.15.28:60602) 2024-09-13T16:44:28.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f7692000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce72ad0 0x55d0fce9c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:51200/0 (socket says 172.21.15.28:51200) 2024-09-13T16:44:28.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f769b400640 1 -- 172.21.15.28:0/1346983653 learned_addr learned my addr 172.21.15.28:0/1346983653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:28.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f7692000640 1 -- 172.21.15.28:0/1346983653 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fceab2d0 msgr2=0x55d0fcea2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:28.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f7692000640 1 --2- 172.21.15.28:0/1346983653 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fceab2d0 0x55d0fcea2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:28.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f7692000640 1 -- 172.21.15.28:0/1346983653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce70300 msgr2=0x55d0fcea05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:28.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f7692000640 1 --2- 172.21.15.28:0/1346983653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce70300 0x55d0fcea05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:28.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f7692000640 1 -- 172.21.15.28:0/1346983653 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7684088040 con 0x55d0fce72ad0 2024-09-13T16:44:28.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.063+0000 7f769aa00640 1 --2- 172.21.15.28:0/1346983653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce70300 0x55d0fcea05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:28.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.064+0000 7f7692000640 1 --2- 172.21.15.28:0/1346983653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce72ad0 0x55d0fce9c6e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7684086d00 tx=0x7f7684000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6715e804024e192 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:28.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.064+0000 7f767fe00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-09-13T16:44:28.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.064+0000 7f767fe00640 1 -- 172.21.15.28:0/1346983653 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7684056c10 con 0x55d0fce72ad0 2024-09-13T16:44:28.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.064+0000 7f767fe00640 1 -- 172.21.15.28:0/1346983653 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7684056d70 con 0x55d0fce72ad0 2024-09-13T16:44:28.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.064+0000 7f76a47fc740 1 -- 172.21.15.28:0/1346983653 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d0fcdfdcd0 con 0x55d0fce72ad0 2024-09-13T16:44:28.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.064+0000 7f767fe00640 1 -- 172.21.15.28:0/1346983653 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f768405f5d0 con 0x55d0fce72ad0 2024-09-13T16:44:28.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.064+0000 7f76a47fc740 10 client.4711 fetch_fsmap learned FSMap version 17 2024-09-13T16:44:28.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.064+0000 7f76a47fc740 1 -- 172.21.15.28:0/1346983653 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55d0fcdd21b0 con 0x55d0fce72ad0 2024-09-13T16:44:28.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.064+0000 7f767fe00640 1 -- 172.21.15.28:0/1346983653 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d0fcdfdcd0 con 0x55d0fce72ad0 2024-09-13T16:44:28.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.064+0000 7f767fe00640 1 -- 172.21.15.28:0/1346983653 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 17) v1 ==== 4252+0+0 (secure 0 0 0) 0x7f768405d440 con 0x55d0fce72ad0 2024-09-13T16:44:28.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.064+0000 7f76a47fc740 10 client.4711 fetch_fsmap finished waiting for FSMap version 17 2024-09-13T16:44:28.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.064+0000 7f76a47fc740 10 client.4711 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-13T16:44:28.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.064+0000 7f76a47fc740 20 client.4711 populate_metadata read hostname 'smithi028' 2024-09-13T16:44:28.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.064+0000 7f76a47fc740 1 --2- 172.21.15.28:0/1346983653 >> [v2:172.21.15.191:6837/2366383263,v1:172.21.15.191:6839/2366383263] conn(0x55d0fcec4d90 0x55d0fcec5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:28.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.065+0000 7f76a47fc740 4 client.4711 mds_command: new command op to 4232 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:44:28.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.065+0000 7f76a47fc740 1 -- 172.21.15.28:0/1346983653 --> [v2:172.21.15.191:6837/2366383263,v1:172.21.15.191:6839/2366383263] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d0fcac73c0 con 0x55d0fcec4d90 2024-09-13T16:44:28.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.065+0000 7f769aa00640 1 --2- 172.21.15.28:0/1346983653 >> [v2:172.21.15.191:6837/2366383263,v1:172.21.15.191:6839/2366383263] conn(0x55d0fcec4d90 0x55d0fcec5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:28.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.066+0000 7f769aa00640 1 --2- 172.21.15.28:0/1346983653 >> [v2:172.21.15.191:6837/2366383263,v1:172.21.15.191:6839/2366383263] conn(0x55d0fcec4d90 0x55d0fcec5170 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f584a0fdd6bc5166 server_cookie=43584c2c2681d465 in_seq=0 out_seq=0 2024-09-13T16:44:28.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:28.066+0000 7f767fe00640 10 client.4711 ms_handle_connect on v2:172.21.15.191:6837/2366383263 2024-09-13T16:44:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.693+0000 7f767fe00640 1 -- 172.21.15.28:0/1346983653 <== mds.4 v2:172.21.15.191:6837/2366383263 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d0fcac73c0 con 0x55d0fcec4d90 2024-09-13T16:44:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.693+0000 7f767fe00640 10 client.4711 handle_command_reply: tid=0 2024-09-13T16:44:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.693+0000 7f76a47fc740 1 client.4711 shutdown 2024-09-13T16:44:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.693+0000 7f76a47fc740 1 -- 172.21.15.28:0/1346983653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce72ad0 msgr2=0x55d0fce9c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.693+0000 7f76a47fc740 1 --2- 172.21.15.28:0/1346983653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce72ad0 0x55d0fce9c6e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7684086d00 tx=0x7f7684000ed0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.693+0000 7f76a47fc740 1 -- 172.21.15.28:0/1346983653 shutdown_connections 2024-09-13T16:44:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.693+0000 7f76a47fc740 1 --2- 172.21.15.28:0/1346983653 >> [v2:172.21.15.191:6837/2366383263,v1:172.21.15.191:6839/2366383263] conn(0x55d0fcec4d90 0x55d0fcec5170 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.693+0000 7f76a47fc740 1 --2- 172.21.15.28:0/1346983653 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fceab2d0 0x55d0fcea2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.693+0000 7f76a47fc740 1 --2- 172.21.15.28:0/1346983653 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce72ad0 0x55d0fce9c6e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.693+0000 7f767fe00640 0 client.4711 ms_handle_reset on v2:172.21.15.191:6837/2366383263 2024-09-13T16:44:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.693+0000 7f76a47fc740 1 --2- 172.21.15.28:0/1346983653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce70300 0x55d0fcea05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.693+0000 7f76a47fc740 1 -- 172.21.15.28:0/1346983653 >> 172.21.15.28:0/1346983653 conn(0x55d0fce711f0 msgr2=0x55d0fce706e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.693+0000 7f76a47fc740 1 -- 172.21.15.28:0/1346983653 shutdown_connections 2024-09-13T16:44:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.693+0000 7f76a47fc740 1 -- 172.21.15.28:0/1346983653 wait complete. 2024-09-13T16:44:31.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.693+0000 7f76a47fc740 20 client.4711 trim_cache size 0 max 16384 2024-09-13T16:44:31.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.710+0000 7f76a47fc740 1 Processor -- start 2024-09-13T16:44:31.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.710+0000 7f76a47fc740 1 -- start start 2024-09-13T16:44:31.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.711+0000 7f76a47fc740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fce70300 0x55d0fcecb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.711+0000 7f76a47fc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce72ad0 0x55d0fce706e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.711+0000 7f76a47fc740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fceab2d0 0x55d0fce70c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.711+0000 7f76a47fc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fce6d650 con 0x55d0fce72ad0 2024-09-13T16:44:31.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.711+0000 7f76a47fc740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fcea25d0 con 0x55d0fceab2d0 2024-09-13T16:44:31.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.711+0000 7f76a47fc740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d0fceaa8d0 con 0x55d0fce70300 2024-09-13T16:44:31.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.711+0000 7f7692000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce72ad0 0x55d0fce706e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.712+0000 7f769aa00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fce70300 0x55d0fcecb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.712+0000 7f769b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fceab2d0 0x55d0fce70c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.712+0000 7f7692000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce72ad0 0x55d0fce706e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44050/0 (socket says 172.21.15.28:44050) 2024-09-13T16:44:31.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.712+0000 7f7692000640 1 -- 172.21.15.28:0/2778067845 learned_addr learned my addr 172.21.15.28:0/2778067845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.712+0000 7f769aa00640 1 -- 172.21.15.28:0/2778067845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fceab2d0 msgr2=0x55d0fce70c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.712+0000 7f769aa00640 1 --2- 172.21.15.28:0/2778067845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fceab2d0 0x55d0fce70c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.712+0000 7f769aa00640 1 -- 172.21.15.28:0/2778067845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce72ad0 msgr2=0x55d0fce706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.712+0000 7f769aa00640 1 --2- 172.21.15.28:0/2778067845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce72ad0 0x55d0fce706e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.712+0000 7f769aa00640 1 -- 172.21.15.28:0/2778067845 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0fce07170 con 0x55d0fce70300 2024-09-13T16:44:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.712+0000 7f769b400640 1 --2- 172.21.15.28:0/2778067845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fceab2d0 0x55d0fce70c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.712+0000 7f769aa00640 1 --2- 172.21.15.28:0/2778067845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fce70300 0x55d0fcecb5c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f768c02a250 tx=0x7f768c02c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17a263edee1406b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:31.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.713+0000 7f768be00640 1 -- 172.21.15.28:0/2778067845 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c04e040 con 0x55d0fce70300 2024-09-13T16:44:31.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.713+0000 7f768be00640 1 -- 172.21.15.28:0/2778067845 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768c050040 con 0x55d0fce70300 2024-09-13T16:44:31.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.713+0000 7f768be00640 1 -- 172.21.15.28:0/2778067845 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c055920 con 0x55d0fce70300 2024-09-13T16:44:31.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.713+0000 7f76a47fc740 1 -- 172.21.15.28:0/2778067845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fce70300 msgr2=0x55d0fcecb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:31.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.713+0000 7f76a47fc740 1 --2- 172.21.15.28:0/2778067845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fce70300 0x55d0fcecb5c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f768c02a250 tx=0x7f768c02c000 comp rx=0 tx=0).stop 2024-09-13T16:44:31.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.714+0000 7f76a47fc740 1 -- 172.21.15.28:0/2778067845 shutdown_connections 2024-09-13T16:44:31.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.714+0000 7f76a47fc740 1 --2- 172.21.15.28:0/2778067845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fceab2d0 0x55d0fce70c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.714+0000 7f76a47fc740 1 --2- 172.21.15.28:0/2778067845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce72ad0 0x55d0fce706e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.714+0000 7f76a47fc740 1 --2- 172.21.15.28:0/2778067845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fce70300 0x55d0fcecb5c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.714+0000 7f76a47fc740 1 -- 172.21.15.28:0/2778067845 >> 172.21.15.28:0/2778067845 conn(0x55d0fce711f0 msgr2=0x55d0fced41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:31.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.714+0000 7f76a47fc740 1 -- 172.21.15.28:0/2778067845 shutdown_connections 2024-09-13T16:44:31.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.714+0000 7f76a47fc740 1 -- 172.21.15.28:0/2778067845 wait complete. 2024-09-13T16:44:31.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.714+0000 7f76a47fc740 1 -- start start 2024-09-13T16:44:31.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.715+0000 7f76a47fc740 1 Processor -- start 2024-09-13T16:44:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.716+0000 7f76a47fc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce70300 0x55d0fcea5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.716+0000 7f76a47fc740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce72ad0 0x55d0fcf15f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.716+0000 7f76a47fc740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fceab2d0 0x55d0fcf16490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.716+0000 7f76a47fc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fced4ac0 con 0x55d0fce70300 2024-09-13T16:44:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.716+0000 7f76a47fc740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fce96930 con 0x55d0fce72ad0 2024-09-13T16:44:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.716+0000 7f76a47fc740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d0fceb4d20 con 0x55d0fceab2d0 2024-09-13T16:44:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.716+0000 7f76a47fc740 10 client.0 _ll_register_callbacks cb 0x7f768406e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:44:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.716+0000 7f769aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce70300 0x55d0fcea5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.716+0000 7f769aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce70300 0x55d0fcea5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44054/0 (socket says 172.21.15.28:44054) 2024-09-13T16:44:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.716+0000 7f7692000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce72ad0 0x55d0fcf15f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.716+0000 7f769aa00640 1 -- 172.21.15.28:0/3890487407 learned_addr learned my addr 172.21.15.28:0/3890487407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.716+0000 7f769b400640 1 --2- 172.21.15.28:0/3890487407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fceab2d0 0x55d0fcf16490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.717+0000 7f769aa00640 1 -- 172.21.15.28:0/3890487407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fceab2d0 msgr2=0x55d0fcf16490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:31.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.717+0000 7f769aa00640 1 --2- 172.21.15.28:0/3890487407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fceab2d0 0x55d0fcf16490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.717+0000 7f769aa00640 1 -- 172.21.15.28:0/3890487407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce72ad0 msgr2=0x55d0fcf15f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.717+0000 7f769aa00640 1 --2- 172.21.15.28:0/3890487407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce72ad0 0x55d0fcf15f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.717+0000 7f769aa00640 1 -- 172.21.15.28:0/3890487407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7694017410 con 0x55d0fce70300 2024-09-13T16:44:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.717+0000 7f769b400640 1 --2- 172.21.15.28:0/3890487407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fceab2d0 0x55d0fcf16490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.717+0000 7f7692000640 1 --2- 172.21.15.28:0/3890487407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce72ad0 0x55d0fcf15f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.717+0000 7f769aa00640 1 --2- 172.21.15.28:0/3890487407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce70300 0x55d0fcea5ec0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f768c0098c0 tx=0x7f768c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc90a7efc7c65ea server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.717+0000 7f767fe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T16:44:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.717+0000 7f767fe00640 1 -- 172.21.15.28:0/3890487407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c04e040 con 0x55d0fce70300 2024-09-13T16:44:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.717+0000 7f767fe00640 1 -- 172.21.15.28:0/3890487407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f768c03c950 con 0x55d0fce70300 2024-09-13T16:44:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.717+0000 7f76a47fc740 1 -- 172.21.15.28:0/3890487407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7684055ac0 con 0x55d0fce70300 2024-09-13T16:44:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.718+0000 7f767fe00640 1 -- 172.21.15.28:0/3890487407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f768c03cb10 con 0x55d0fce70300 2024-09-13T16:44:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.718+0000 7f76a47fc740 10 client.4632 fetch_fsmap learned FSMap version 17 2024-09-13T16:44:31.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.718+0000 7f76a47fc740 1 -- 172.21.15.28:0/3890487407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x7f768c02e070 con 0x55d0fce70300 2024-09-13T16:44:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.718+0000 7f767fe00640 1 -- 172.21.15.28:0/3890487407 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7684055ac0 con 0x55d0fce70300 2024-09-13T16:44:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.718+0000 7f767fe00640 1 -- 172.21.15.28:0/3890487407 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 17) v1 ==== 4252+0+0 (secure 0 0 0) 0x7f768c05d030 con 0x55d0fce70300 2024-09-13T16:44:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.718+0000 7f76a47fc740 10 client.4632 fetch_fsmap finished waiting for FSMap version 17 2024-09-13T16:44:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.718+0000 7f76a47fc740 10 client.4632 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-13T16:44:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.718+0000 7f76a47fc740 20 client.4632 populate_metadata read hostname 'smithi028' 2024-09-13T16:44:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.718+0000 7f76a47fc740 1 --2- 172.21.15.28:0/3890487407 >> [v2:172.21.15.191:6837/2366383263,v1:172.21.15.191:6839/2366383263] conn(0x55d0fce96150 0x55d0fcebe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.718+0000 7f76a47fc740 4 client.4632 mds_command: new command op to 4232 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:44:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.718+0000 7f76a47fc740 1 -- 172.21.15.28:0/3890487407 --> [v2:172.21.15.191:6837/2366383263,v1:172.21.15.191:6839/2366383263] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d0fcad86c0 con 0x55d0fce96150 2024-09-13T16:44:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.719+0000 7f7692000640 1 --2- 172.21.15.28:0/3890487407 >> [v2:172.21.15.191:6837/2366383263,v1:172.21.15.191:6839/2366383263] conn(0x55d0fce96150 0x55d0fcebe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.719+0000 7f7692000640 1 --2- 172.21.15.28:0/3890487407 >> [v2:172.21.15.191:6837/2366383263,v1:172.21.15.191:6839/2366383263] conn(0x55d0fce96150 0x55d0fcebe0d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=877697035790865c server_cookie=41faa22c5370aefe in_seq=0 out_seq=0 2024-09-13T16:44:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.719+0000 7f767fe00640 10 client.4632 ms_handle_connect on v2:172.21.15.191:6837/2366383263 2024-09-13T16:44:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.719+0000 7f767fe00640 1 -- 172.21.15.28:0/3890487407 <== mds.4 v2:172.21.15.191:6837/2366383263 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55d0fcad86c0 con 0x55d0fce96150 2024-09-13T16:44:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.719+0000 7f767fe00640 10 client.4632 handle_command_reply: tid=0 2024-09-13T16:44:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.719+0000 7f76a47fc740 1 client.4632 shutdown 2024-09-13T16:44:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.719+0000 7f76a47fc740 1 -- 172.21.15.28:0/3890487407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce70300 msgr2=0x55d0fcea5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.719+0000 7f76a47fc740 1 --2- 172.21.15.28:0/3890487407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce70300 0x55d0fcea5ec0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f768c0098c0 tx=0x7f768c004730 comp rx=0 tx=0).stop 2024-09-13T16:44:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.720+0000 7f76a47fc740 1 -- 172.21.15.28:0/3890487407 shutdown_connections 2024-09-13T16:44:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.720+0000 7f76a47fc740 1 --2- 172.21.15.28:0/3890487407 >> [v2:172.21.15.191:6837/2366383263,v1:172.21.15.191:6839/2366383263] conn(0x55d0fce96150 0x55d0fcebe0d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.720+0000 7f76a47fc740 1 --2- 172.21.15.28:0/3890487407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d0fceab2d0 0x55d0fcf16490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.720+0000 7f76a47fc740 1 --2- 172.21.15.28:0/3890487407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d0fce72ad0 0x55d0fcf15f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.720+0000 7f76a47fc740 1 --2- 172.21.15.28:0/3890487407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0fce70300 0x55d0fcea5ec0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.720+0000 7f767fe00640 0 client.4632 ms_handle_reset on v2:172.21.15.191:6837/2366383263 2024-09-13T16:44:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.720+0000 7f76a47fc740 1 -- 172.21.15.28:0/3890487407 >> 172.21.15.28:0/3890487407 conn(0x55d0fce711f0 msgr2=0x55d0fced41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.720+0000 7f76a47fc740 1 -- 172.21.15.28:0/3890487407 shutdown_connections 2024-09-13T16:44:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.720+0000 7f76a47fc740 1 -- 172.21.15.28:0/3890487407 wait complete. 2024-09-13T16:44:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.720+0000 7f76a47fc740 20 client.4632 trim_cache size 0 max 16384 2024-09-13T16:44:31.725 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:44:31.725 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:44:31.725 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 4, 2024-09-13T16:44:31.725 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4232, 2024-09-13T16:44:31.725 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T16:44:31.725 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T16:44:31.725 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T16:44:31.725 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 11.049835404, 2024-09-13T16:44:31.725 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 17, 2024-09-13T16:44:31.725 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 20, 2024-09-13T16:44:31.726 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 20, 2024-09-13T16:44:31.726 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 41.498661646000002 2024-09-13T16:44:31.726 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:44:31.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.721+0000 7f76a1a00640 1 -- 172.21.15.28:0/947809723 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f767806ac00 msgr2=0x7f767806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.721+0000 7f76a1a00640 1 --2- 172.21.15.28:0/947809723 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f767806ac00 0x7f767806d0c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f768c034000 tx=0x7f768c02e000 comp rx=0 tx=0).stop 2024-09-13T16:44:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.721+0000 7f76a1a00640 1 -- 172.21.15.28:0/947809723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f769c0c5bd0 msgr2=0x7f769c1617b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.721+0000 7f76a1a00640 1 --2- 172.21.15.28:0/947809723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f769c0c5bd0 0x7f769c1617b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f768400c530 tx=0x7f768400c9f0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.721+0000 7f76a1a00640 1 -- 172.21.15.28:0/947809723 shutdown_connections 2024-09-13T16:44:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.721+0000 7f76a1a00640 1 --2- 172.21.15.28:0/947809723 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f767806ac00 0x7f767806d0c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.721+0000 7f76a1a00640 1 --2- 172.21.15.28:0/947809723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f769c0c6ef0 0x7f769c161cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.721+0000 7f76a1a00640 1 --2- 172.21.15.28:0/947809723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f769c0c5bd0 0x7f769c1617b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.721+0000 7f76a1a00640 1 --2- 172.21.15.28:0/947809723 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f769c0c3670 0x7f769c14c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.721+0000 7f76a1a00640 1 -- 172.21.15.28:0/947809723 >> 172.21.15.28:0/947809723 conn(0x7f769c066040 msgr2=0x7f769c070890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.721+0000 7f76a1a00640 1 -- 172.21.15.28:0/947809723 shutdown_connections 2024-09-13T16:44:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.721+0000 7f76a1a00640 1 -- 172.21.15.28:0/947809723 wait complete. 2024-09-13T16:44:31.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-09-13T16:44:31.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.900+0000 7fe254600640 1 Processor -- start 2024-09-13T16:44:31.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.900+0000 7fe254600640 1 -- start start 2024-09-13T16:44:31.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.901+0000 7fe254600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe24c0c5bd0 0x7fe24c156150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.901+0000 7fe254600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe24c0c6ef0 0x7fe24c0c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.901+0000 7fe254600640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24c14ca00 con 0x7fe24c0c3670 2024-09-13T16:44:31.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.901+0000 7fe254600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24c0c2dc0 con 0x7fe24c0c6ef0 2024-09-13T16:44:31.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.901+0000 7fe254600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c155780 con 0x7fe24c0c5bd0 2024-09-13T16:44:31.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.901+0000 7fe252600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe24c0c6ef0 0x7fe24c0c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.901+0000 7fe251c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe24c0c5bd0 0x7fe24c156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.901+0000 7fe251200640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fe24c0c3670 0x7fe24c0c1960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:42718/0 (socket says 172.21.15.28:42718) 2024-09-13T16:44:31.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.901+0000 7fe251200640 1 -- 172.21.15.28:0/3812317539 learned_addr learned my addr 172.21.15.28:0/3812317539 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:31.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.902+0000 7fe251c00640 1 -- 172.21.15.28:0/3812317539 >> v2:172.21.15.28:3300/0 conn(0x7fe24c0c6ef0 msgr2=0x7fe24c0c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:31.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.902+0000 7fe251c00640 1 --2- 172.21.15.28:0/3812317539 >> v2:172.21.15.28:3300/0 conn(0x7fe24c0c6ef0 0x7fe24c0c7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.902+0000 7fe251c00640 1 -- 172.21.15.28:0/3812317539 >> v1:172.21.15.191:6789/0 conn(0x7fe24c0c3670 legacy=0x7fe24c0c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:31.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.902+0000 7fe251c00640 1 -- 172.21.15.28:0/3812317539 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24c155990 con 0x7fe24c0c5bd0 2024-09-13T16:44:31.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.903+0000 7fe251c00640 1 --2- 172.21.15.28:0/3812317539 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe24c0c5bd0 0x7fe24c156150 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe2400097a0 tx=0x7fe24002a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91942d4c68059e7c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:31.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.903+0000 7fe24be00640 1 -- 172.21.15.28:0/3812317539 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe240032440 con 0x7fe24c0c5bd0 2024-09-13T16:44:31.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.903+0000 7fe24be00640 1 -- 172.21.15.28:0/3812317539 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2400325a0 con 0x7fe24c0c5bd0 2024-09-13T16:44:31.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.903+0000 7fe24be00640 1 -- 172.21.15.28:0/3812317539 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe240032930 con 0x7fe24c0c5bd0 2024-09-13T16:44:31.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.904+0000 7fe254600640 1 -- 172.21.15.28:0/3812317539 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe24c0c5bd0 msgr2=0x7fe24c156150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:31.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.904+0000 7fe254600640 1 --2- 172.21.15.28:0/3812317539 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe24c0c5bd0 0x7fe24c156150 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe2400097a0 tx=0x7fe24002a230 comp rx=0 tx=0).stop 2024-09-13T16:44:31.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.904+0000 7fe254600640 1 -- 172.21.15.28:0/3812317539 shutdown_connections 2024-09-13T16:44:31.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.904+0000 7fe254600640 1 --2- 172.21.15.28:0/3812317539 >> v2:172.21.15.28:3300/0 conn(0x7fe24c0c6ef0 0x7fe24c0c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.904+0000 7fe254600640 1 --2- 172.21.15.28:0/3812317539 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe24c0c5bd0 0x7fe24c156150 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.904+0000 7fe254600640 1 -- 172.21.15.28:0/3812317539 >> 172.21.15.28:0/3812317539 conn(0x7fe24c066040 msgr2=0x7fe24c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:31.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.904+0000 7fe254600640 1 -- 172.21.15.28:0/3812317539 shutdown_connections 2024-09-13T16:44:31.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.904+0000 7fe254600640 1 -- 172.21.15.28:0/3812317539 wait complete. 2024-09-13T16:44:31.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.905+0000 7fe254600640 1 Processor -- start 2024-09-13T16:44:31.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.905+0000 7fe254600640 1 -- start start 2024-09-13T16:44:31.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.906+0000 7fe254600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24c0c3670 0x7fe24c14e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.906+0000 7fe254600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe24c0c5bd0 0x7fe24c142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.906+0000 7fe254600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe24c0c6ef0 0x7fe24c142f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.906+0000 7fe254600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c1ee5f0 con 0x7fe24c0c3670 2024-09-13T16:44:31.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.906+0000 7fe254600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c157430 con 0x7fe24c0c6ef0 2024-09-13T16:44:31.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.906+0000 7fe254600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c157710 con 0x7fe24c0c5bd0 2024-09-13T16:44:31.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.906+0000 7fe251c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24c0c3670 0x7fe24c14e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.906+0000 7fe251c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24c0c3670 0x7fe24c14e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44086/0 (socket says 172.21.15.28:44086) 2024-09-13T16:44:31.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.906+0000 7fe251c00640 1 -- 172.21.15.28:0/2312053686 learned_addr learned my addr 172.21.15.28:0/2312053686 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:31.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.906+0000 7fe251200640 1 --2- 172.21.15.28:0/2312053686 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe24c0c5bd0 0x7fe24c142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.906+0000 7fe252600640 1 --2- 172.21.15.28:0/2312053686 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe24c0c6ef0 0x7fe24c142f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.907+0000 7fe251c00640 1 -- 172.21.15.28:0/2312053686 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe24c0c5bd0 msgr2=0x7fe24c142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:31.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.907+0000 7fe251c00640 1 --2- 172.21.15.28:0/2312053686 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe24c0c5bd0 0x7fe24c142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.907+0000 7fe251c00640 1 -- 172.21.15.28:0/2312053686 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe24c0c6ef0 msgr2=0x7fe24c142f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:31.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.907+0000 7fe251c00640 1 --2- 172.21.15.28:0/2312053686 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe24c0c6ef0 0x7fe24c142f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.907+0000 7fe251c00640 1 -- 172.21.15.28:0/2312053686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe240009400 con 0x7fe24c0c3670 2024-09-13T16:44:31.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.907+0000 7fe251c00640 1 --2- 172.21.15.28:0/2312053686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24c0c3670 0x7fe24c14e0f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe240035040 tx=0x7fe2400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:31.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.907+0000 7fe249600640 1 -- 172.21.15.28:0/2312053686 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe240006e10 con 0x7fe24c0c3670 2024-09-13T16:44:31.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.907+0000 7fe249600640 1 -- 172.21.15.28:0/2312053686 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe240032e90 con 0x7fe24c0c3670 2024-09-13T16:44:31.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.907+0000 7fe254600640 1 -- 172.21.15.28:0/2312053686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe24c1579f0 con 0x7fe24c0c3670 2024-09-13T16:44:31.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.907+0000 7fe249600640 1 -- 172.21.15.28:0/2312053686 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe240030be0 con 0x7fe24c0c3670 2024-09-13T16:44:31.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.908+0000 7fe254600640 1 -- 172.21.15.28:0/2312053686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe24c14e630 con 0x7fe24c0c3670 2024-09-13T16:44:31.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.909+0000 7fe249600640 1 -- 172.21.15.28:0/2312053686 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe240039040 con 0x7fe24c0c3670 2024-09-13T16:44:31.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.910+0000 7fe249600640 1 --2- 172.21.15.28:0/2312053686 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe22c06ac00 0x7fe22c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.910+0000 7fe249600640 1 -- 172.21.15.28:0/2312053686 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2400a9a70 con 0x7fe24c0c3670 2024-09-13T16:44:31.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.911+0000 7fe251200640 1 --2- 172.21.15.28:0/2312053686 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe22c06ac00 0x7fe22c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.911+0000 7fe25747c740 1 Processor -- start 2024-09-13T16:44:31.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.911+0000 7fe25747c740 1 -- start start 2024-09-13T16:44:31.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.911+0000 7fe251200640 1 --2- 172.21.15.28:0/2312053686 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe22c06ac00 0x7fe22c06d0c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe24c0c47f0 tx=0x7fe238009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:31.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.912+0000 7fe25747c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8753ad0 0x5605e878bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.912+0000 7fe25747c740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8751300 0x5605e874fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.912+0000 7fe25747c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e878c2d0 0x5605e8750320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.912+0000 7fe25747c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605e86bbab0 con 0x5605e878c2d0 2024-09-13T16:44:31.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.912+0000 7fe25747c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605e86b9240 con 0x5605e8753ad0 2024-09-13T16:44:31.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.912+0000 7fe25747c740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x5605e86b4b30 con 0x5605e8751300 2024-09-13T16:44:31.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.912+0000 7fe252600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e878c2d0 0x5605e8750320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.912+0000 7fe252600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e878c2d0 0x5605e8750320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44100/0 (socket says 172.21.15.28:44100) 2024-09-13T16:44:31.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.912+0000 7fe252600640 1 -- 172.21.15.28:0/2420593793 learned_addr learned my addr 172.21.15.28:0/2420593793 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:31.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.912+0000 7fe251200640 1 --2- 172.21.15.28:0/2420593793 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8751300 0x5605e874fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.912+0000 7fe251c00640 1 --2- 172.21.15.28:0/2420593793 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8753ad0 0x5605e878bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.912+0000 7fe252600640 1 -- 172.21.15.28:0/2420593793 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8751300 msgr2=0x5605e874fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:31.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.912+0000 7fe252600640 1 --2- 172.21.15.28:0/2420593793 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8751300 0x5605e874fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.912+0000 7fe252600640 1 -- 172.21.15.28:0/2420593793 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8753ad0 msgr2=0x5605e878bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:31.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.912+0000 7fe252600640 1 --2- 172.21.15.28:0/2420593793 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8753ad0 0x5605e878bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.912+0000 7fe252600640 1 -- 172.21.15.28:0/2420593793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605e867fbd0 con 0x5605e878c2d0 2024-09-13T16:44:31.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.912+0000 7fe251c00640 1 --2- 172.21.15.28:0/2420593793 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8753ad0 0x5605e878bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:31.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.913+0000 7fe252600640 1 --2- 172.21.15.28:0/2420593793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e878c2d0 0x5605e8750320 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe244011580 tx=0x7fe244009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5868863b0b214d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:31.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.913+0000 7fe23e000640 1 -- 172.21.15.28:0/2420593793 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24400e040 con 0x5605e878c2d0 2024-09-13T16:44:31.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.913+0000 7fe23e000640 1 -- 172.21.15.28:0/2420593793 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24400f040 con 0x5605e878c2d0 2024-09-13T16:44:31.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.913+0000 7fe23e000640 1 -- 172.21.15.28:0/2420593793 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe244015a40 con 0x5605e878c2d0 2024-09-13T16:44:31.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.913+0000 7fe25747c740 1 -- 172.21.15.28:0/2420593793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e878c2d0 msgr2=0x5605e8750320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:31.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.913+0000 7fe25747c740 1 --2- 172.21.15.28:0/2420593793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e878c2d0 0x5605e8750320 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe244011580 tx=0x7fe244009750 comp rx=0 tx=0).stop 2024-09-13T16:44:31.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.914+0000 7fe25747c740 1 -- 172.21.15.28:0/2420593793 shutdown_connections 2024-09-13T16:44:31.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.914+0000 7fe25747c740 1 --2- 172.21.15.28:0/2420593793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e878c2d0 0x5605e8750320 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.914+0000 7fe25747c740 1 --2- 172.21.15.28:0/2420593793 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8751300 0x5605e874fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.914+0000 7fe25747c740 1 --2- 172.21.15.28:0/2420593793 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8753ad0 0x5605e878bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.914+0000 7fe25747c740 1 -- 172.21.15.28:0/2420593793 >> 172.21.15.28:0/2420593793 conn(0x5605e87521f0 msgr2=0x5605e87525f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:31.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.914+0000 7fe25747c740 1 -- 172.21.15.28:0/2420593793 shutdown_connections 2024-09-13T16:44:31.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.914+0000 7fe25747c740 1 -- 172.21.15.28:0/2420593793 wait complete. 2024-09-13T16:44:31.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.914+0000 7fe25747c740 1 -- start start 2024-09-13T16:44:31.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.915+0000 7fe25747c740 1 Processor -- start 2024-09-13T16:44:31.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.915+0000 7fe25747c740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8751300 0x5605e87839f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.915+0000 7fe25747c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e8753ad0 0x5605e877d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.915+0000 7fe25747c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e878c2d0 0x5605e877d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.915+0000 7fe25747c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605e8656260 con 0x5605e8753ad0 2024-09-13T16:44:31.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.915+0000 7fe25747c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605e8655f00 con 0x5605e878c2d0 2024-09-13T16:44:31.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.915+0000 7fe25747c740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x5605e841bf20 con 0x5605e8751300 2024-09-13T16:44:31.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.915+0000 7fe25747c740 10 client.0 _ll_register_callbacks cb 0x5605e86ae1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:44:31.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.916+0000 7fe251200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e8753ad0 0x5605e877d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.916+0000 7fe251c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8751300 0x5605e87839f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.916+0000 7fe251200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e8753ad0 0x5605e877d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44114/0 (socket says 172.21.15.28:44114) 2024-09-13T16:44:31.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.916+0000 7fe251200640 1 -- 172.21.15.28:0/3427913477 learned_addr learned my addr 172.21.15.28:0/3427913477 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:31.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.916+0000 7fe252600640 1 --2- 172.21.15.28:0/3427913477 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e878c2d0 0x5605e877d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.916+0000 7fe251200640 1 -- 172.21.15.28:0/3427913477 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8751300 msgr2=0x5605e87839f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:31.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.916+0000 7fe251200640 1 --2- 172.21.15.28:0/3427913477 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8751300 0x5605e87839f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.916+0000 7fe251200640 1 -- 172.21.15.28:0/3427913477 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e878c2d0 msgr2=0x5605e877d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:31.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.916+0000 7fe251200640 1 --2- 172.21.15.28:0/3427913477 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e878c2d0 0x5605e877d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.916+0000 7fe251200640 1 -- 172.21.15.28:0/3427913477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe244011230 con 0x5605e8753ad0 2024-09-13T16:44:31.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.916+0000 7fe251c00640 1 --2- 172.21.15.28:0/3427913477 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8751300 0x5605e87839f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:31.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.916+0000 7fe251200640 1 --2- 172.21.15.28:0/3427913477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e8753ad0 0x5605e877d260 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe238014680 tx=0x7fe2380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=594a4880d2945a7a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:31.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.916+0000 7fe232a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T16:44:31.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.917+0000 7fe232a00640 1 -- 172.21.15.28:0/3427913477 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe238013740 con 0x5605e8753ad0 2024-09-13T16:44:31.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.917+0000 7fe232a00640 1 -- 172.21.15.28:0/3427913477 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe23801c040 con 0x5605e8753ad0 2024-09-13T16:44:31.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.917+0000 7fe25747c740 1 -- 172.21.15.28:0/3427913477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5605e86decd0 con 0x5605e8753ad0 2024-09-13T16:44:31.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.917+0000 7fe232a00640 1 -- 172.21.15.28:0/3427913477 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2380138a0 con 0x5605e8753ad0 2024-09-13T16:44:31.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.917+0000 7fe25747c740 10 client.4644 fetch_fsmap learned FSMap version 17 2024-09-13T16:44:31.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.917+0000 7fe25747c740 1 -- 172.21.15.28:0/3427913477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5605e86b31b0 con 0x5605e8753ad0 2024-09-13T16:44:31.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.917+0000 7fe232a00640 1 -- 172.21.15.28:0/3427913477 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5605e86decd0 con 0x5605e8753ad0 2024-09-13T16:44:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.917+0000 7fe232a00640 1 -- 172.21.15.28:0/3427913477 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 17) v1 ==== 4252+0+0 (secure 0 0 0) 0x7fe238012070 con 0x5605e8753ad0 2024-09-13T16:44:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.917+0000 7fe25747c740 10 client.4644 fetch_fsmap finished waiting for FSMap version 17 2024-09-13T16:44:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.917+0000 7fe25747c740 10 client.4644 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-13T16:44:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.917+0000 7fe25747c740 20 client.4644 populate_metadata read hostname 'smithi028' 2024-09-13T16:44:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.917+0000 7fe25747c740 1 --2- 172.21.15.28:0/3427913477 >> [v2:172.21.15.191:6836/2702860767,v1:172.21.15.191:6838/2702860767] conn(0x5605e87a6020 0x5605e87a6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.917+0000 7fe25747c740 4 client.4644 mds_command: new command op to 4235 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:44:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.917+0000 7fe25747c740 1 -- 172.21.15.28:0/3427913477 --> [v2:172.21.15.191:6836/2702860767,v1:172.21.15.191:6838/2702860767] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5605e83a83c0 con 0x5605e87a6020 2024-09-13T16:44:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.918+0000 7fe251c00640 1 --2- 172.21.15.28:0/3427913477 >> [v2:172.21.15.191:6836/2702860767,v1:172.21.15.191:6838/2702860767] conn(0x5605e87a6020 0x5605e87a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.918+0000 7fe251c00640 1 --2- 172.21.15.28:0/3427913477 >> [v2:172.21.15.191:6836/2702860767,v1:172.21.15.191:6838/2702860767] conn(0x5605e87a6020 0x5605e87a6400 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d3edaf6a637b9e29 server_cookie=2c5568b705469aac in_seq=0 out_seq=0 2024-09-13T16:44:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.918+0000 7fe232a00640 10 client.4644 ms_handle_connect on v2:172.21.15.191:6836/2702860767 2024-09-13T16:44:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.919+0000 7fe232a00640 1 -- 172.21.15.28:0/3427913477 <== mds.3 v2:172.21.15.191:6836/2702860767 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x7fe2400a1e90 con 0x5605e87a6020 2024-09-13T16:44:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.919+0000 7fe232a00640 10 client.4644 handle_command_reply: tid=0 2024-09-13T16:44:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.919+0000 7fe25747c740 1 client.4644 shutdown 2024-09-13T16:44:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.919+0000 7fe25747c740 1 -- 172.21.15.28:0/3427913477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e8753ad0 msgr2=0x5605e877d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.919+0000 7fe25747c740 1 --2- 172.21.15.28:0/3427913477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e8753ad0 0x5605e877d260 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe238014680 tx=0x7fe2380057f0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.919+0000 7fe25747c740 1 -- 172.21.15.28:0/3427913477 shutdown_connections 2024-09-13T16:44:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.919+0000 7fe25747c740 1 --2- 172.21.15.28:0/3427913477 >> [v2:172.21.15.191:6836/2702860767,v1:172.21.15.191:6838/2702860767] conn(0x5605e87a6020 0x5605e87a6400 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.919+0000 7fe25747c740 1 --2- 172.21.15.28:0/3427913477 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e878c2d0 0x5605e877d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.919+0000 7fe25747c740 1 --2- 172.21.15.28:0/3427913477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e8753ad0 0x5605e877d260 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.919+0000 7fe25747c740 1 --2- 172.21.15.28:0/3427913477 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8751300 0x5605e87839f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.919+0000 7fe232a00640 0 client.4644 ms_handle_reset on v2:172.21.15.191:6836/2702860767 2024-09-13T16:44:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.919+0000 7fe25747c740 1 -- 172.21.15.28:0/3427913477 >> 172.21.15.28:0/3427913477 conn(0x5605e87521f0 msgr2=0x5605e8751700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.919+0000 7fe25747c740 1 -- 172.21.15.28:0/3427913477 shutdown_connections 2024-09-13T16:44:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.919+0000 7fe25747c740 1 -- 172.21.15.28:0/3427913477 wait complete. 2024-09-13T16:44:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.920+0000 7fe25747c740 20 client.4644 trim_cache size 0 max 16384 2024-09-13T16:44:31.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.935+0000 7fe25747c740 1 Processor -- start 2024-09-13T16:44:31.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.935+0000 7fe25747c740 1 -- start start 2024-09-13T16:44:31.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.936+0000 7fe25747c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8751300 0x5605e87ac640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.936+0000 7fe25747c740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8753ad0 0x5605e87516e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.936+0000 7fe25747c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e878c2d0 0x5605e8751c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.936+0000 7fe25747c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605e874dd50 con 0x5605e878c2d0 2024-09-13T16:44:31.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.936+0000 7fe25747c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605e87a5960 con 0x5605e8751300 2024-09-13T16:44:31.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.936+0000 7fe25747c740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x5605e8750fe0 con 0x5605e8753ad0 2024-09-13T16:44:31.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.936+0000 7fe252600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e878c2d0 0x5605e8751c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.936+0000 7fe251200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8753ad0 0x5605e87516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.936+0000 7fe251c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8751300 0x5605e87ac640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.936+0000 7fe251200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8753ad0 0x5605e87516e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:55218/0 (socket says 172.21.15.28:55218) 2024-09-13T16:44:31.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.936+0000 7fe251200640 1 -- 172.21.15.28:0/3627590372 learned_addr learned my addr 172.21.15.28:0/3627590372 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.937+0000 7fe251200640 1 -- 172.21.15.28:0/3627590372 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8751300 msgr2=0x5605e87ac640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.937+0000 7fe251200640 1 --2- 172.21.15.28:0/3627590372 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8751300 0x5605e87ac640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.937+0000 7fe251200640 1 -- 172.21.15.28:0/3627590372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e878c2d0 msgr2=0x5605e8751c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.937+0000 7fe251200640 1 --2- 172.21.15.28:0/3627590372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e878c2d0 0x5605e8751c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.937+0000 7fe251200640 1 -- 172.21.15.28:0/3627590372 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605e874c510 con 0x5605e8753ad0 2024-09-13T16:44:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.937+0000 7fe251c00640 1 --2- 172.21.15.28:0/3627590372 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8751300 0x5605e87ac640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.937+0000 7fe251200640 1 --2- 172.21.15.28:0/3627590372 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8753ad0 0x5605e87516e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe2380017e0 tx=0x7fe238021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70db54fadbe4bd90 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.937+0000 7fe23ea00640 1 -- 172.21.15.28:0/3627590372 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe238013740 con 0x5605e8753ad0 2024-09-13T16:44:31.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.937+0000 7fe23ea00640 1 -- 172.21.15.28:0/3627590372 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2380138a0 con 0x5605e8753ad0 2024-09-13T16:44:31.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.937+0000 7fe23ea00640 1 -- 172.21.15.28:0/3627590372 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2380060d0 con 0x5605e8753ad0 2024-09-13T16:44:31.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.938+0000 7fe25747c740 1 -- 172.21.15.28:0/3627590372 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8753ad0 msgr2=0x5605e87516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:31.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.938+0000 7fe25747c740 1 --2- 172.21.15.28:0/3627590372 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8753ad0 0x5605e87516e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe2380017e0 tx=0x7fe238021000 comp rx=0 tx=0).stop 2024-09-13T16:44:31.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.938+0000 7fe25747c740 1 -- 172.21.15.28:0/3627590372 shutdown_connections 2024-09-13T16:44:31.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.938+0000 7fe25747c740 1 --2- 172.21.15.28:0/3627590372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e878c2d0 0x5605e8751c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.938+0000 7fe25747c740 1 --2- 172.21.15.28:0/3627590372 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e8753ad0 0x5605e87516e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.938+0000 7fe25747c740 1 --2- 172.21.15.28:0/3627590372 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8751300 0x5605e87ac640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.938+0000 7fe25747c740 1 -- 172.21.15.28:0/3627590372 >> 172.21.15.28:0/3627590372 conn(0x5605e87521f0 msgr2=0x5605e8752d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:31.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.938+0000 7fe25747c740 1 -- 172.21.15.28:0/3627590372 shutdown_connections 2024-09-13T16:44:31.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.938+0000 7fe25747c740 1 -- 172.21.15.28:0/3627590372 wait complete. 2024-09-13T16:44:31.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.938+0000 7fe25747c740 1 -- start start 2024-09-13T16:44:31.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.939+0000 7fe25747c740 1 Processor -- start 2024-09-13T16:44:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.939+0000 7fe25747c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e8751300 0x5605e878efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.939+0000 7fe25747c740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8753ad0 0x5605e8781e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe25747c740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e878c2d0 0x5605e87f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe25747c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605e8752940 con 0x5605e8751300 2024-09-13T16:44:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe25747c740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605e874ea60 con 0x5605e8753ad0 2024-09-13T16:44:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe25747c740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x5605e8752b50 con 0x5605e878c2d0 2024-09-13T16:44:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe25747c740 10 client.0 _ll_register_callbacks cb 0x7fe2380015a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe251c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e8751300 0x5605e878efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe251200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8753ad0 0x5605e8781e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe251c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e8751300 0x5605e878efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44130/0 (socket says 172.21.15.28:44130) 2024-09-13T16:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe251200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8753ad0 0x5605e8781e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:50618/0 (socket says 172.21.15.28:50618) 2024-09-13T16:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe251c00640 1 -- 172.21.15.28:0/1154107506 learned_addr learned my addr 172.21.15.28:0/1154107506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe252600640 1 --2- 172.21.15.28:0/1154107506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e878c2d0 0x5605e87f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe251c00640 1 -- 172.21.15.28:0/1154107506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e878c2d0 msgr2=0x5605e87f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe251c00640 1 --2- 172.21.15.28:0/1154107506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e878c2d0 0x5605e87f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe251c00640 1 -- 172.21.15.28:0/1154107506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8753ad0 msgr2=0x5605e8781e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe251c00640 1 --2- 172.21.15.28:0/1154107506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8753ad0 0x5605e8781e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe251c00640 1 -- 172.21.15.28:0/1154107506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605e86b31b0 con 0x5605e8751300 2024-09-13T16:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe251c00640 1 --2- 172.21.15.28:0/1154107506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e8751300 0x5605e878efd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe2400a7f90 tx=0x7fe240008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66373806cda33678 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.940+0000 7fe232a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T16:44:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.941+0000 7fe232a00640 1 -- 172.21.15.28:0/1154107506 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2400903b0 con 0x5605e8751300 2024-09-13T16:44:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.941+0000 7fe232a00640 1 -- 172.21.15.28:0/1154107506 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe240078d30 con 0x5605e8751300 2024-09-13T16:44:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.941+0000 7fe25747c740 1 -- 172.21.15.28:0/1154107506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe24400f040 con 0x5605e8751300 2024-09-13T16:44:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.941+0000 7fe232a00640 1 -- 172.21.15.28:0/1154107506 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe240078ef0 con 0x5605e8751300 2024-09-13T16:44:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.941+0000 7fe25747c740 10 client.4650 fetch_fsmap learned FSMap version 17 2024-09-13T16:44:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.941+0000 7fe25747c740 1 -- 172.21.15.28:0/1154107506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5605e86e8170 con 0x5605e8751300 2024-09-13T16:44:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.941+0000 7fe232a00640 1 -- 172.21.15.28:0/1154107506 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24400f040 con 0x5605e8751300 2024-09-13T16:44:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.941+0000 7fe232a00640 1 -- 172.21.15.28:0/1154107506 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 17) v1 ==== 4252+0+0 (secure 0 0 0) 0x7fe240077fc0 con 0x5605e8751300 2024-09-13T16:44:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.941+0000 7fe25747c740 10 client.4650 fetch_fsmap finished waiting for FSMap version 17 2024-09-13T16:44:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.941+0000 7fe25747c740 10 client.4650 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-13T16:44:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.941+0000 7fe25747c740 20 client.4650 populate_metadata read hostname 'smithi028' 2024-09-13T16:44:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.941+0000 7fe25747c740 1 --2- 172.21.15.28:0/1154107506 >> [v2:172.21.15.191:6836/2702860767,v1:172.21.15.191:6838/2702860767] conn(0x5605e87f7740 0x5605e8800870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.941+0000 7fe25747c740 4 client.4650 mds_command: new command op to 4235 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:44:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.941+0000 7fe25747c740 1 -- 172.21.15.28:0/1154107506 --> [v2:172.21.15.191:6836/2702860767,v1:172.21.15.191:6838/2702860767] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5605e83b96c0 con 0x5605e87f7740 2024-09-13T16:44:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.941+0000 7fe251200640 1 --2- 172.21.15.28:0/1154107506 >> [v2:172.21.15.191:6836/2702860767,v1:172.21.15.191:6838/2702860767] conn(0x5605e87f7740 0x5605e8800870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.942+0000 7fe251200640 1 --2- 172.21.15.28:0/1154107506 >> [v2:172.21.15.191:6836/2702860767,v1:172.21.15.191:6838/2702860767] conn(0x5605e87f7740 0x5605e8800870 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c2614acaf968d78e server_cookie=b3c910765b8e2b5c in_seq=0 out_seq=0 2024-09-13T16:44:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.942+0000 7fe232a00640 10 client.4650 ms_handle_connect on v2:172.21.15.191:6836/2702860767 2024-09-13T16:44:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.942+0000 7fe232a00640 1 -- 172.21.15.28:0/1154107506 <== mds.3 v2:172.21.15.191:6836/2702860767 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5605e83b96c0 con 0x5605e87f7740 2024-09-13T16:44:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.942+0000 7fe232a00640 10 client.4650 handle_command_reply: tid=0 2024-09-13T16:44:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.942+0000 7fe25747c740 1 client.4650 shutdown 2024-09-13T16:44:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.942+0000 7fe25747c740 1 -- 172.21.15.28:0/1154107506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e8751300 msgr2=0x5605e878efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.942+0000 7fe25747c740 1 --2- 172.21.15.28:0/1154107506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e8751300 0x5605e878efd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe2400a7f90 tx=0x7fe240008040 comp rx=0 tx=0).stop 2024-09-13T16:44:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.942+0000 7fe25747c740 1 -- 172.21.15.28:0/1154107506 shutdown_connections 2024-09-13T16:44:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.942+0000 7fe25747c740 1 --2- 172.21.15.28:0/1154107506 >> [v2:172.21.15.191:6836/2702860767,v1:172.21.15.191:6838/2702860767] conn(0x5605e87f7740 0x5605e8800870 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.942+0000 7fe25747c740 1 --2- 172.21.15.28:0/1154107506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5605e878c2d0 0x5605e87f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.942+0000 7fe25747c740 1 --2- 172.21.15.28:0/1154107506 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5605e8753ad0 0x5605e8781e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.942+0000 7fe25747c740 1 --2- 172.21.15.28:0/1154107506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605e8751300 0x5605e878efd0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.942+0000 7fe232a00640 0 client.4650 ms_handle_reset on v2:172.21.15.191:6836/2702860767 2024-09-13T16:44:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.942+0000 7fe25747c740 1 -- 172.21.15.28:0/1154107506 >> 172.21.15.28:0/1154107506 conn(0x5605e87521f0 msgr2=0x5605e8752d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.943+0000 7fe25747c740 1 -- 172.21.15.28:0/1154107506 shutdown_connections 2024-09-13T16:44:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.943+0000 7fe25747c740 1 -- 172.21.15.28:0/1154107506 wait complete. 2024-09-13T16:44:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.943+0000 7fe25747c740 20 client.4650 trim_cache size 0 max 16384 2024-09-13T16:44:31.949 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:44:31.949 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:44:31.949 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 3, 2024-09-13T16:44:31.950 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4235, 2024-09-13T16:44:31.950 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T16:44:31.950 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T16:44:31.950 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T16:44:31.950 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 17.278858897999999, 2024-09-13T16:44:31.950 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 17, 2024-09-13T16:44:31.950 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 20, 2024-09-13T16:44:31.950 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 20, 2024-09-13T16:44:31.950 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 41.721653801999999 2024-09-13T16:44:31.950 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:44:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.943+0000 7fe254600640 1 -- 172.21.15.28:0/2312053686 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe22c06ac00 msgr2=0x7fe22c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.943+0000 7fe254600640 1 --2- 172.21.15.28:0/2312053686 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe22c06ac00 0x7fe22c06d0c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe24c0c47f0 tx=0x7fe238009040 comp rx=0 tx=0).stop 2024-09-13T16:44:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.943+0000 7fe254600640 1 -- 172.21.15.28:0/2312053686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24c0c3670 msgr2=0x7fe24c14e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.943+0000 7fe254600640 1 --2- 172.21.15.28:0/2312053686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24c0c3670 0x7fe24c14e0f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe240035040 tx=0x7fe2400046c0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.943+0000 7fe254600640 1 -- 172.21.15.28:0/2312053686 shutdown_connections 2024-09-13T16:44:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.943+0000 7fe254600640 1 --2- 172.21.15.28:0/2312053686 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe22c06ac00 0x7fe22c06d0c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.943+0000 7fe254600640 1 --2- 172.21.15.28:0/2312053686 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe24c0c6ef0 0x7fe24c142f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.943+0000 7fe254600640 1 --2- 172.21.15.28:0/2312053686 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe24c0c5bd0 0x7fe24c142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.943+0000 7fe254600640 1 --2- 172.21.15.28:0/2312053686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24c0c3670 0x7fe24c14e0f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.943+0000 7fe254600640 1 -- 172.21.15.28:0/2312053686 >> 172.21.15.28:0/2312053686 conn(0x7fe24c066040 msgr2=0x7fe24c14bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.943+0000 7fe254600640 1 -- 172.21.15.28:0/2312053686 shutdown_connections 2024-09-13T16:44:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:31.943+0000 7fe254600640 1 -- 172.21.15.28:0/2312053686 wait complete. 2024-09-13T16:44:31.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T16:44:32.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.113+0000 7f8024e00640 1 Processor -- start 2024-09-13T16:44:32.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.114+0000 7f8024e00640 1 -- start start 2024-09-13T16:44:32.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.115+0000 7f8024e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8020150d70 0x7f80200c17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.115+0000 7f8024e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80200c4550 0x7f80200c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.115+0000 7f8024e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80200c64f0 con 0x7f8020150990 2024-09-13T16:44:32.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.115+0000 7f8024e00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80200c7410 con 0x7f8020150d70 2024-09-13T16:44:32.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.115+0000 7f8024e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80200c76d0 con 0x7f80200c4550 2024-09-13T16:44:32.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.115+0000 7f801f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8020150990 0x7f8020156870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53760/0 (socket says 172.21.15.28:53760) 2024-09-13T16:44:32.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.115+0000 7f801f400640 1 -- 172.21.15.28:0/3652829337 learned_addr learned my addr 172.21.15.28:0/3652829337 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:32.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.115+0000 7f8016a00640 1 --2- 172.21.15.28:0/3652829337 >> v2:172.21.15.191:3300/0 conn(0x7f8020150d70 0x7f80200c17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.115+0000 7f801fe00640 1 --2- 172.21.15.28:0/3652829337 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80200c4550 0x7f80200c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.115+0000 7f801ea00640 1 -- 172.21.15.28:0/3652829337 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1475342708 0 0) 0x7f80200c64f0 con 0x7f8020150990 2024-09-13T16:44:32.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.116+0000 7f801fe00640 1 -- 172.21.15.28:0/3652829337 >> v2:172.21.15.191:3300/0 conn(0x7f8020150d70 msgr2=0x7f80200c17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.116+0000 7f801fe00640 1 --2- 172.21.15.28:0/3652829337 >> v2:172.21.15.191:3300/0 conn(0x7f8020150d70 0x7f80200c17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.116+0000 7f801fe00640 1 -- 172.21.15.28:0/3652829337 >> v1:172.21.15.28:6789/0 conn(0x7f8020150990 legacy=0x7f8020156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.116+0000 7f801fe00640 1 -- 172.21.15.28:0/3652829337 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80200c5ea0 con 0x7f80200c4550 2024-09-13T16:44:32.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.116+0000 7f8016a00640 1 --2- 172.21.15.28:0/3652829337 >> v2:172.21.15.191:3300/0 conn(0x7f8020150d70 0x7f80200c17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:32.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.117+0000 7f801fe00640 1 --2- 172.21.15.28:0/3652829337 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80200c4550 0x7f80200c6d80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8018009d90 tx=0x7f801802a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d958107463e3ce9b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.117+0000 7f801ea00640 1 -- 172.21.15.28:0/3652829337 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8018007740 con 0x7f80200c4550 2024-09-13T16:44:32.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.117+0000 7f801ea00640 1 -- 172.21.15.28:0/3652829337 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801802d050 con 0x7f80200c4550 2024-09-13T16:44:32.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.117+0000 7f801ea00640 1 -- 172.21.15.28:0/3652829337 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8018007ac0 con 0x7f80200c4550 2024-09-13T16:44:32.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.117+0000 7f8024e00640 1 -- 172.21.15.28:0/3652829337 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80200c4550 msgr2=0x7f80200c6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.117+0000 7f8024e00640 1 --2- 172.21.15.28:0/3652829337 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80200c4550 0x7f80200c6d80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8018009d90 tx=0x7f801802a6f0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.117+0000 7f8024e00640 1 -- 172.21.15.28:0/3652829337 shutdown_connections 2024-09-13T16:44:32.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.117+0000 7f8024e00640 1 --2- 172.21.15.28:0/3652829337 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f80200c4550 0x7f80200c6d80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.117+0000 7f8024e00640 1 --2- 172.21.15.28:0/3652829337 >> v2:172.21.15.191:3300/0 conn(0x7f8020150d70 0x7f80200c17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.117+0000 7f8024e00640 1 -- 172.21.15.28:0/3652829337 >> 172.21.15.28:0/3652829337 conn(0x7f8020069e80 msgr2=0x7f802006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.118+0000 7f8024e00640 1 -- 172.21.15.28:0/3652829337 shutdown_connections 2024-09-13T16:44:32.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.118+0000 7f8024e00640 1 -- 172.21.15.28:0/3652829337 wait complete. 2024-09-13T16:44:32.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.119+0000 7f8024e00640 1 Processor -- start 2024-09-13T16:44:32.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.119+0000 7f8024e00640 1 -- start start 2024-09-13T16:44:32.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.120+0000 7f8024e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80200c4550 0x7f80200bdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.120+0000 7f8024e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8020150990 0x7f80200caa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.120+0000 7f8024e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8020150d70 0x7f802014b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.120+0000 7f8024e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80200be750 con 0x7f80200c4550 2024-09-13T16:44:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.120+0000 7f8024e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80201ee170 con 0x7f8020150990 2024-09-13T16:44:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.120+0000 7f8024e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80201eef90 con 0x7f8020150d70 2024-09-13T16:44:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.120+0000 7f801f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80200c4550 0x7f80200bdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.120+0000 7f801f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80200c4550 0x7f80200bdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44132/0 (socket says 172.21.15.28:44132) 2024-09-13T16:44:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.120+0000 7f801f400640 1 -- 172.21.15.28:0/37944867 learned_addr learned my addr 172.21.15.28:0/37944867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.120+0000 7f8016a00640 1 --2- 172.21.15.28:0/37944867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8020150990 0x7f80200caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.120+0000 7f801fe00640 1 --2- 172.21.15.28:0/37944867 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8020150d70 0x7f802014b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.121+0000 7f801f400640 1 -- 172.21.15.28:0/37944867 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8020150d70 msgr2=0x7f802014b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.121+0000 7f801f400640 1 --2- 172.21.15.28:0/37944867 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8020150d70 0x7f802014b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.121+0000 7f801f400640 1 -- 172.21.15.28:0/37944867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8020150990 msgr2=0x7f80200caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.121+0000 7f801f400640 1 --2- 172.21.15.28:0/37944867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8020150990 0x7f80200caa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.121+0000 7f801f400640 1 -- 172.21.15.28:0/37944867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8008009400 con 0x7f80200c4550 2024-09-13T16:44:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.121+0000 7f801f400640 1 --2- 172.21.15.28:0/37944867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80200c4550 0x7f80200bdba0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f801000c4d0 tx=0x7f801000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.121+0000 7f8017e00640 1 -- 172.21.15.28:0/37944867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80100074e0 con 0x7f80200c4550 2024-09-13T16:44:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.121+0000 7f8017e00640 1 -- 172.21.15.28:0/37944867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8010007660 con 0x7f80200c4550 2024-09-13T16:44:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.121+0000 7f8017e00640 1 -- 172.21.15.28:0/37944867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010004a20 con 0x7f80200c4550 2024-09-13T16:44:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.121+0000 7f8024e00640 1 -- 172.21.15.28:0/37944867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8018009a70 con 0x7f80200c4550 2024-09-13T16:44:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.122+0000 7f8024e00640 1 -- 172.21.15.28:0/37944867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80201ee380 con 0x7f80200c4550 2024-09-13T16:44:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.124+0000 7f8017e00640 1 -- 172.21.15.28:0/37944867 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8010004b80 con 0x7f80200c4550 2024-09-13T16:44:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.124+0000 7f8017e00640 1 --2- 172.21.15.28:0/37944867 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ffc06ac00 0x7f7ffc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.124+0000 7f8017e00640 1 -- 172.21.15.28:0/37944867 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8010088510 con 0x7f80200c4550 2024-09-13T16:44:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.125+0000 7f8016a00640 1 --2- 172.21.15.28:0/37944867 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ffc06ac00 0x7f7ffc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.125+0000 7f802926a740 1 Processor -- start 2024-09-13T16:44:32.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.125+0000 7f802926a740 1 -- start start 2024-09-13T16:44:32.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.125+0000 7f8016a00640 1 --2- 172.21.15.28:0/37944867 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ffc06ac00 0x7f7ffc06d0c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f802014d580 tx=0x7f8008009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.126+0000 7f802926a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e4424c40 0x5618e445cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.126+0000 7f802926a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e4422470 0x5618e4420f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.126+0000 7f802926a740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e445d440 0x5618e4421490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.126+0000 7f802926a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618e438cab0 con 0x5618e4424c40 2024-09-13T16:44:32.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.126+0000 7f802926a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618e438a240 con 0x5618e4422470 2024-09-13T16:44:32.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.126+0000 7f802926a740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x5618e4385b30 con 0x5618e445d440 2024-09-13T16:44:32.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.126+0000 7f801f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e4424c40 0x5618e445cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.126+0000 7f801f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e4424c40 0x5618e445cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44146/0 (socket says 172.21.15.28:44146) 2024-09-13T16:44:32.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.126+0000 7f801f400640 1 -- 172.21.15.28:0/2618604439 learned_addr learned my addr 172.21.15.28:0/2618604439 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:32.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.126+0000 7f8016a00640 1 --2- 172.21.15.28:0/2618604439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e4422470 0x5618e4420f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.126+0000 7f801fe00640 1 --2- 172.21.15.28:0/2618604439 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e445d440 0x5618e4421490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.126+0000 7f801f400640 1 -- 172.21.15.28:0/2618604439 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e445d440 msgr2=0x5618e4421490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.126+0000 7f801f400640 1 --2- 172.21.15.28:0/2618604439 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e445d440 0x5618e4421490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.126+0000 7f801f400640 1 -- 172.21.15.28:0/2618604439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e4422470 msgr2=0x5618e4420f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.126+0000 7f801f400640 1 --2- 172.21.15.28:0/2618604439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e4422470 0x5618e4420f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.126+0000 7f801f400640 1 -- 172.21.15.28:0/2618604439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618e4350bd0 con 0x5618e4424c40 2024-09-13T16:44:32.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.127+0000 7f801f400640 1 --2- 172.21.15.28:0/2618604439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e4424c40 0x5618e445cf00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f80100538c0 tx=0x7f8010053800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f133da3141c465b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.127+0000 7f800fe00640 1 -- 172.21.15.28:0/2618604439 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801005f700 con 0x5618e4424c40 2024-09-13T16:44:32.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.127+0000 7f800fe00640 1 -- 172.21.15.28:0/2618604439 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8010055500 con 0x5618e4424c40 2024-09-13T16:44:32.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.127+0000 7f800fe00640 1 -- 172.21.15.28:0/2618604439 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010053ad0 con 0x5618e4424c40 2024-09-13T16:44:32.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.127+0000 7f802926a740 1 -- 172.21.15.28:0/2618604439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e4424c40 msgr2=0x5618e445cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.127+0000 7f802926a740 1 --2- 172.21.15.28:0/2618604439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e4424c40 0x5618e445cf00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f80100538c0 tx=0x7f8010053800 comp rx=0 tx=0).stop 2024-09-13T16:44:32.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.128+0000 7f802926a740 1 -- 172.21.15.28:0/2618604439 shutdown_connections 2024-09-13T16:44:32.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.128+0000 7f802926a740 1 --2- 172.21.15.28:0/2618604439 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e445d440 0x5618e4421490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.128+0000 7f802926a740 1 --2- 172.21.15.28:0/2618604439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e4422470 0x5618e4420f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.128+0000 7f802926a740 1 --2- 172.21.15.28:0/2618604439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e4424c40 0x5618e445cf00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.128+0000 7f802926a740 1 -- 172.21.15.28:0/2618604439 >> 172.21.15.28:0/2618604439 conn(0x5618e4423360 msgr2=0x5618e4423760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.128+0000 7f802926a740 1 -- 172.21.15.28:0/2618604439 shutdown_connections 2024-09-13T16:44:32.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.128+0000 7f802926a740 1 -- 172.21.15.28:0/2618604439 wait complete. 2024-09-13T16:44:32.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.128+0000 7f802926a740 1 -- start start 2024-09-13T16:44:32.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.128+0000 7f802926a740 1 Processor -- start 2024-09-13T16:44:32.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.129+0000 7f802926a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e4422470 0x5618e44537b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.129+0000 7f802926a740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e4424c40 0x5618e4466360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.129+0000 7f802926a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e445d440 0x5618e444e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.129+0000 7f802926a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618e4327260 con 0x5618e445d440 2024-09-13T16:44:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.129+0000 7f802926a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618e4326f00 con 0x5618e4422470 2024-09-13T16:44:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.129+0000 7f802926a740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x5618e40ecf20 con 0x5618e4424c40 2024-09-13T16:44:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.129+0000 7f802926a740 10 client.0 _ll_register_callbacks cb 0x5618e437f1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:44:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.130+0000 7f801f400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e4422470 0x5618e44537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.130+0000 7f8016a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e4424c40 0x5618e4466360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.130+0000 7f801f400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e4422470 0x5618e44537b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:50656/0 (socket says 172.21.15.28:50656) 2024-09-13T16:44:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.130+0000 7f801fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e445d440 0x5618e444e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.130+0000 7f801f400640 1 -- 172.21.15.28:0/2836243059 learned_addr learned my addr 172.21.15.28:0/2836243059 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.130+0000 7f801f400640 1 -- 172.21.15.28:0/2836243059 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e4424c40 msgr2=0x5618e4466360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.130+0000 7f801f400640 1 --2- 172.21.15.28:0/2836243059 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e4424c40 0x5618e4466360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.130+0000 7f801f400640 1 -- 172.21.15.28:0/2836243059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e445d440 msgr2=0x5618e444e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.130+0000 7f801f400640 1 --2- 172.21.15.28:0/2836243059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e445d440 0x5618e444e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.130+0000 7f801f400640 1 -- 172.21.15.28:0/2836243059 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80201ee380 con 0x5618e4422470 2024-09-13T16:44:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.130+0000 7f801fe00640 1 --2- 172.21.15.28:0/2836243059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e445d440 0x5618e444e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.130+0000 7f801f400640 1 --2- 172.21.15.28:0/2836243059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e4422470 0x5618e44537b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f80100538c0 tx=0x7f8010088050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2045077ca6cacc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.130+0000 7f800cc00640 10 client.0 ms_handle_connect on v2:172.21.15.191:3300/0 2024-09-13T16:44:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.130+0000 7f800cc00640 1 -- 172.21.15.28:0/2836243059 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801005f700 con 0x5618e4422470 2024-09-13T16:44:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.131+0000 7f800cc00640 1 -- 172.21.15.28:0/2836243059 <== mon.1 v2:172.21.15.191:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80100882d0 con 0x5618e4422470 2024-09-13T16:44:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.131+0000 7f802926a740 1 -- 172.21.15.28:0/2836243059 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5618e43afcd0 con 0x5618e4422470 2024-09-13T16:44:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.131+0000 7f800cc00640 1 -- 172.21.15.28:0/2836243059 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f801005f270 con 0x5618e4422470 2024-09-13T16:44:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.131+0000 7f802926a740 10 client.4747 fetch_fsmap learned FSMap version 17 2024-09-13T16:44:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.131+0000 7f802926a740 1 -- 172.21.15.28:0/2836243059 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5618e43841b0 con 0x5618e4422470 2024-09-13T16:44:32.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.131+0000 7f800cc00640 1 -- 172.21.15.28:0/2836243059 <== mon.1 v2:172.21.15.191:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5618e43afcd0 con 0x5618e4422470 2024-09-13T16:44:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.131+0000 7f800cc00640 1 -- 172.21.15.28:0/2836243059 <== mon.1 v2:172.21.15.191:3300/0 5 ==== fsmap(e 17) v1 ==== 4252+0+0 (secure 0 0 0) 0x7f801005d100 con 0x5618e4422470 2024-09-13T16:44:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.131+0000 7f802926a740 10 client.4747 fetch_fsmap finished waiting for FSMap version 17 2024-09-13T16:44:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.131+0000 7f802926a740 10 client.4747 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:44:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.131+0000 7f802926a740 20 client.4747 populate_metadata read hostname 'smithi028' 2024-09-13T16:44:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.131+0000 7f802926a740 1 --2- 172.21.15.28:0/2836243059 >> [v2:172.21.15.28:6832/2240589617,v1:172.21.15.28:6833/2240589617] conn(0x5618e444f5c0 0x5618e4421470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.131+0000 7f802926a740 4 client.4747 mds_command: new command op to 4244 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:44:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.131+0000 7f802926a740 1 -- 172.21.15.28:0/2836243059 --> [v2:172.21.15.28:6832/2240589617,v1:172.21.15.28:6833/2240589617] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5618e40793c0 con 0x5618e444f5c0 2024-09-13T16:44:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.131+0000 7f8016a00640 1 --2- 172.21.15.28:0/2836243059 >> [v2:172.21.15.28:6832/2240589617,v1:172.21.15.28:6833/2240589617] conn(0x5618e444f5c0 0x5618e4421470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.132+0000 7f8016a00640 1 --2- 172.21.15.28:0/2836243059 >> [v2:172.21.15.28:6832/2240589617,v1:172.21.15.28:6833/2240589617] conn(0x5618e444f5c0 0x5618e4421470 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=17281dcdb8d960f4 server_cookie=fe69475e69f7d3ea in_seq=0 out_seq=0 2024-09-13T16:44:32.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.132+0000 7f800cc00640 10 client.4747 ms_handle_connect on v2:172.21.15.28:6832/2240589617 2024-09-13T16:44:32.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.133+0000 7f800cc00640 1 -- 172.21.15.28:0/2836243059 <== mds.2 v2:172.21.15.28:6832/2240589617 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5618e40793c0 con 0x5618e444f5c0 2024-09-13T16:44:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.133+0000 7f800cc00640 10 client.4747 handle_command_reply: tid=0 2024-09-13T16:44:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.133+0000 7f802926a740 1 client.4747 shutdown 2024-09-13T16:44:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.133+0000 7f802926a740 1 -- 172.21.15.28:0/2836243059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e4422470 msgr2=0x5618e44537b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.133+0000 7f802926a740 1 --2- 172.21.15.28:0/2836243059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e4422470 0x5618e44537b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f80100538c0 tx=0x7f8010088050 comp rx=0 tx=0).stop 2024-09-13T16:44:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.133+0000 7f802926a740 1 -- 172.21.15.28:0/2836243059 shutdown_connections 2024-09-13T16:44:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.133+0000 7f802926a740 1 --2- 172.21.15.28:0/2836243059 >> [v2:172.21.15.28:6832/2240589617,v1:172.21.15.28:6833/2240589617] conn(0x5618e444f5c0 0x5618e4421470 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.133+0000 7f802926a740 1 --2- 172.21.15.28:0/2836243059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e445d440 0x5618e444e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.133+0000 7f802926a740 1 --2- 172.21.15.28:0/2836243059 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e4424c40 0x5618e4466360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.133+0000 7f802926a740 1 --2- 172.21.15.28:0/2836243059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e4422470 0x5618e44537b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.133+0000 7f802926a740 1 -- 172.21.15.28:0/2836243059 >> 172.21.15.28:0/2836243059 conn(0x5618e4423360 msgr2=0x5618e445da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.133+0000 7f800cc00640 0 client.4747 ms_handle_reset on v2:172.21.15.28:6832/2240589617 2024-09-13T16:44:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.133+0000 7f802926a740 1 -- 172.21.15.28:0/2836243059 shutdown_connections 2024-09-13T16:44:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.133+0000 7f802926a740 1 -- 172.21.15.28:0/2836243059 wait complete. 2024-09-13T16:44:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.134+0000 7f802926a740 20 client.4747 trim_cache size 0 max 16384 2024-09-13T16:44:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.149+0000 7f802926a740 1 Processor -- start 2024-09-13T16:44:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.149+0000 7f802926a740 1 -- start start 2024-09-13T16:44:32.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.150+0000 7f802926a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e4422470 0x5618e441f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.150+0000 7f802926a740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e4424c40 0x5618e441f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.150+0000 7f802926a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e445d440 0x5618e4421470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.150+0000 7f802926a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618e444f2f0 con 0x5618e4422470 2024-09-13T16:44:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.150+0000 7f802926a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618e441fcd0 con 0x5618e445d440 2024-09-13T16:44:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.150+0000 7f802926a740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x5618e441fed0 con 0x5618e4424c40 2024-09-13T16:44:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.150+0000 7f801f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e4422470 0x5618e441f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.150+0000 7f801f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e4422470 0x5618e441f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44158/0 (socket says 172.21.15.28:44158) 2024-09-13T16:44:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.150+0000 7f801f400640 1 -- 172.21.15.28:0/3952759615 learned_addr learned my addr 172.21.15.28:0/3952759615 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.150+0000 7f801f400640 1 -- 172.21.15.28:0/3952759615 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e4424c40 msgr2=0x5618e441f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.150+0000 7f8016a00640 1 --2- 172.21.15.28:0/3952759615 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e4424c40 0x5618e441f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.150+0000 7f801fe00640 1 --2- 172.21.15.28:0/3952759615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e445d440 0x5618e4421470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.150+0000 7f801f400640 1 --2- 172.21.15.28:0/3952759615 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e4424c40 0x5618e441f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.150+0000 7f801f400640 1 -- 172.21.15.28:0/3952759615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e445d440 msgr2=0x5618e4421470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.150+0000 7f801f400640 1 --2- 172.21.15.28:0/3952759615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e445d440 0x5618e4421470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.150+0000 7f801f400640 1 -- 172.21.15.28:0/3952759615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618e43b9170 con 0x5618e4422470 2024-09-13T16:44:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.151+0000 7f801f400640 1 --2- 172.21.15.28:0/3952759615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e4422470 0x5618e441f1a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f80100868e0 tx=0x7f8010006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83261afc68d512e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.151+0000 7f8014c00640 1 -- 172.21.15.28:0/3952759615 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801000e040 con 0x5618e4422470 2024-09-13T16:44:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.151+0000 7f8014c00640 1 -- 172.21.15.28:0/3952759615 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801005f700 con 0x5618e4422470 2024-09-13T16:44:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.151+0000 7f8014c00640 1 -- 172.21.15.28:0/3952759615 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80100573c0 con 0x5618e4422470 2024-09-13T16:44:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.151+0000 7f802926a740 1 -- 172.21.15.28:0/3952759615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e4422470 msgr2=0x5618e441f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.151+0000 7f802926a740 1 --2- 172.21.15.28:0/3952759615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e4422470 0x5618e441f1a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f80100868e0 tx=0x7f8010006000 comp rx=0 tx=0).stop 2024-09-13T16:44:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.151+0000 7f802926a740 1 -- 172.21.15.28:0/3952759615 shutdown_connections 2024-09-13T16:44:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.151+0000 7f802926a740 1 --2- 172.21.15.28:0/3952759615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e445d440 0x5618e4421470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.151+0000 7f802926a740 1 --2- 172.21.15.28:0/3952759615 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e4424c40 0x5618e441f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.151+0000 7f802926a740 1 --2- 172.21.15.28:0/3952759615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e4422470 0x5618e441f1a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.151+0000 7f802926a740 1 -- 172.21.15.28:0/3952759615 >> 172.21.15.28:0/3952759615 conn(0x5618e4423360 msgr2=0x5618e4419550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.152+0000 7f802926a740 1 -- 172.21.15.28:0/3952759615 shutdown_connections 2024-09-13T16:44:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.152+0000 7f802926a740 1 -- 172.21.15.28:0/3952759615 wait complete. 2024-09-13T16:44:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.152+0000 7f802926a740 1 -- start start 2024-09-13T16:44:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.152+0000 7f802926a740 1 Processor -- start 2024-09-13T16:44:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.153+0000 7f802926a740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e4422470 0x5618e44537d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.153+0000 7f802926a740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e4424c40 0x5618e4464e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.153+0000 7f802926a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e445d440 0x5618e4460d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.153+0000 7f802926a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618e4457410 con 0x5618e445d440 2024-09-13T16:44:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.153+0000 7f802926a740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618e44d1640 con 0x5618e4422470 2024-09-13T16:44:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.153+0000 7f802926a740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x5618e4456e90 con 0x5618e4424c40 2024-09-13T16:44:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.153+0000 7f802926a740 10 client.0 _ll_register_callbacks cb 0x5618e4477d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:44:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.153+0000 7f801fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e445d440 0x5618e4460d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.153+0000 7f801fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e445d440 0x5618e4460d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44164/0 (socket says 172.21.15.28:44164) 2024-09-13T16:44:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.153+0000 7f801fe00640 1 -- 172.21.15.28:0/777185803 learned_addr learned my addr 172.21.15.28:0/777185803 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.153+0000 7f8016a00640 1 --2- 172.21.15.28:0/777185803 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e4424c40 0x5618e4464e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.153+0000 7f801f400640 1 --2- 172.21.15.28:0/777185803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e4422470 0x5618e44537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.154+0000 7f801fe00640 1 -- 172.21.15.28:0/777185803 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e4424c40 msgr2=0x5618e4464e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.154+0000 7f801fe00640 1 --2- 172.21.15.28:0/777185803 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e4424c40 0x5618e4464e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.154+0000 7f801fe00640 1 -- 172.21.15.28:0/777185803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e4422470 msgr2=0x5618e44537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.154+0000 7f801fe00640 1 --2- 172.21.15.28:0/777185803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e4422470 0x5618e44537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.154+0000 7f801fe00640 1 -- 172.21.15.28:0/777185803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5618e43841b0 con 0x5618e445d440 2024-09-13T16:44:32.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.154+0000 7f801fe00640 1 --2- 172.21.15.28:0/777185803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e445d440 0x5618e4460d30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f801802ace0 tx=0x7f801802c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=569c13ee67522de1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.154+0000 7f800cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T16:44:32.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.154+0000 7f800cc00640 1 -- 172.21.15.28:0/777185803 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801803c930 con 0x5618e445d440 2024-09-13T16:44:32.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.154+0000 7f800cc00640 1 -- 172.21.15.28:0/777185803 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f801803ca90 con 0x5618e445d440 2024-09-13T16:44:32.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.154+0000 7f802926a740 1 -- 172.21.15.28:0/777185803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8010055500 con 0x5618e445d440 2024-09-13T16:44:32.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.155+0000 7f800cc00640 1 -- 172.21.15.28:0/777185803 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f801803cc50 con 0x5618e445d440 2024-09-13T16:44:32.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.155+0000 7f802926a740 10 client.4668 fetch_fsmap learned FSMap version 17 2024-09-13T16:44:32.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.155+0000 7f802926a740 1 -- 172.21.15.28:0/777185803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5618e4326470 con 0x5618e445d440 2024-09-13T16:44:32.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.155+0000 7f800cc00640 1 -- 172.21.15.28:0/777185803 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8010055500 con 0x5618e445d440 2024-09-13T16:44:32.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.155+0000 7f800cc00640 1 -- 172.21.15.28:0/777185803 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 17) v1 ==== 4252+0+0 (secure 0 0 0) 0x7f8018038040 con 0x5618e445d440 2024-09-13T16:44:32.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.155+0000 7f802926a740 10 client.4668 fetch_fsmap finished waiting for FSMap version 17 2024-09-13T16:44:32.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.155+0000 7f802926a740 10 client.4668 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:44:32.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.155+0000 7f802926a740 20 client.4668 populate_metadata read hostname 'smithi028' 2024-09-13T16:44:32.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.155+0000 7f802926a740 1 --2- 172.21.15.28:0/777185803 >> [v2:172.21.15.28:6832/2240589617,v1:172.21.15.28:6833/2240589617] conn(0x5618e441df30 0x5618e441f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.155+0000 7f802926a740 4 client.4668 mds_command: new command op to 4244 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:44:32.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.155+0000 7f802926a740 1 -- 172.21.15.28:0/777185803 --> [v2:172.21.15.28:6832/2240589617,v1:172.21.15.28:6833/2240589617] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5618e408a6c0 con 0x5618e441df30 2024-09-13T16:44:32.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.155+0000 7f801f400640 1 --2- 172.21.15.28:0/777185803 >> [v2:172.21.15.28:6832/2240589617,v1:172.21.15.28:6833/2240589617] conn(0x5618e441df30 0x5618e441f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.156+0000 7f801f400640 1 --2- 172.21.15.28:0/777185803 >> [v2:172.21.15.28:6832/2240589617,v1:172.21.15.28:6833/2240589617] conn(0x5618e441df30 0x5618e441f390 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4b578b8c13b84e59 server_cookie=ebecbb59d650fa27 in_seq=0 out_seq=0 2024-09-13T16:44:32.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.156+0000 7f800cc00640 10 client.4668 ms_handle_connect on v2:172.21.15.28:6832/2240589617 2024-09-13T16:44:32.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.156+0000 7f800cc00640 1 -- 172.21.15.28:0/777185803 <== mds.2 v2:172.21.15.28:6832/2240589617 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5618e408a6c0 con 0x5618e441df30 2024-09-13T16:44:32.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.156+0000 7f800cc00640 10 client.4668 handle_command_reply: tid=0 2024-09-13T16:44:32.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.156+0000 7f802926a740 1 client.4668 shutdown 2024-09-13T16:44:32.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.156+0000 7f802926a740 1 -- 172.21.15.28:0/777185803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e445d440 msgr2=0x5618e4460d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.156+0000 7f802926a740 1 --2- 172.21.15.28:0/777185803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e445d440 0x5618e4460d30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f801802ace0 tx=0x7f801802c000 comp rx=0 tx=0).stop 2024-09-13T16:44:32.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.156+0000 7f802926a740 1 -- 172.21.15.28:0/777185803 shutdown_connections 2024-09-13T16:44:32.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.156+0000 7f802926a740 1 --2- 172.21.15.28:0/777185803 >> [v2:172.21.15.28:6832/2240589617,v1:172.21.15.28:6833/2240589617] conn(0x5618e441df30 0x5618e441f390 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.156+0000 7f802926a740 1 --2- 172.21.15.28:0/777185803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618e445d440 0x5618e4460d30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.156+0000 7f802926a740 1 --2- 172.21.15.28:0/777185803 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5618e4424c40 0x5618e4464e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.156+0000 7f802926a740 1 --2- 172.21.15.28:0/777185803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5618e4422470 0x5618e44537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.156+0000 7f800cc00640 0 client.4668 ms_handle_reset on v2:172.21.15.28:6832/2240589617 2024-09-13T16:44:32.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.156+0000 7f802926a740 1 -- 172.21.15.28:0/777185803 >> 172.21.15.28:0/777185803 conn(0x5618e4423360 msgr2=0x5618e4473e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.157+0000 7f802926a740 1 -- 172.21.15.28:0/777185803 shutdown_connections 2024-09-13T16:44:32.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.157+0000 7f802926a740 1 -- 172.21.15.28:0/777185803 wait complete. 2024-09-13T16:44:32.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.157+0000 7f802926a740 20 client.4668 trim_cache size 0 max 16384 2024-09-13T16:44:32.162 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:44:32.163 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:44:32.163 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-09-13T16:44:32.163 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4244, 2024-09-13T16:44:32.163 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T16:44:32.163 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T16:44:32.163 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T16:44:32.163 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 23.073876641999998, 2024-09-13T16:44:32.163 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 17, 2024-09-13T16:44:32.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 20, 2024-09-13T16:44:32.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 20, 2024-09-13T16:44:32.163 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 41.887475551999998 2024-09-13T16:44:32.163 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:44:32.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.157+0000 7f8024e00640 1 -- 172.21.15.28:0/37944867 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ffc06ac00 msgr2=0x7f7ffc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.157+0000 7f8024e00640 1 --2- 172.21.15.28:0/37944867 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ffc06ac00 0x7f7ffc06d0c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f802014d580 tx=0x7f8008009190 comp rx=0 tx=0).stop 2024-09-13T16:44:32.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.157+0000 7f8024e00640 1 -- 172.21.15.28:0/37944867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80200c4550 msgr2=0x7f80200bdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.157+0000 7f8024e00640 1 --2- 172.21.15.28:0/37944867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80200c4550 0x7f80200bdba0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f801000c4d0 tx=0x7f801000c990 comp rx=0 tx=0).stop 2024-09-13T16:44:32.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.157+0000 7f8024e00640 1 -- 172.21.15.28:0/37944867 shutdown_connections 2024-09-13T16:44:32.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.157+0000 7f8024e00640 1 --2- 172.21.15.28:0/37944867 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ffc06ac00 0x7f7ffc06d0c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.157+0000 7f8024e00640 1 --2- 172.21.15.28:0/37944867 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8020150d70 0x7f802014b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.157+0000 7f8024e00640 1 --2- 172.21.15.28:0/37944867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8020150990 0x7f80200caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.157+0000 7f8024e00640 1 --2- 172.21.15.28:0/37944867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80200c4550 0x7f80200bdba0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.157+0000 7f8024e00640 1 -- 172.21.15.28:0/37944867 >> 172.21.15.28:0/37944867 conn(0x7f8020069e80 msgr2=0x7f802014f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.158+0000 7f8024e00640 1 -- 172.21.15.28:0/37944867 shutdown_connections 2024-09-13T16:44:32.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.158+0000 7f8024e00640 1 -- 172.21.15.28:0/37944867 wait complete. 2024-09-13T16:44:32.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-09-13T16:44:32.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.327+0000 7f5b6b200640 1 Processor -- start 2024-09-13T16:44:32.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.328+0000 7f5b6b200640 1 -- start start 2024-09-13T16:44:32.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.329+0000 7f5b6b200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b6c150990 0x7f5b6c0c17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.329+0000 7f5b6b200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5b6c0c4550 0x7f5b6c0c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.329+0000 7f5b6b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b6c0c64f0 con 0x7f5b6c150d70 2024-09-13T16:44:32.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.329+0000 7f5b6b200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6c0c7410 con 0x7f5b6c0c4550 2024-09-13T16:44:32.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.329+0000 7f5b6b200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c0c76d0 con 0x7f5b6c150990 2024-09-13T16:44:32.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.329+0000 7f5b61400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5b6c150d70 0x7f5b6c156870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53762/0 (socket says 172.21.15.28:53762) 2024-09-13T16:44:32.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.329+0000 7f5b61400640 1 -- 172.21.15.28:0/4062428869 learned_addr learned my addr 172.21.15.28:0/4062428869 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:32.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.329+0000 7f5b69e00640 1 --2- 172.21.15.28:0/4062428869 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b6c150990 0x7f5b6c0c17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.329+0000 7f5b6a800640 1 --2- 172.21.15.28:0/4062428869 >> v2:172.21.15.191:3300/0 conn(0x7f5b6c0c4550 0x7f5b6c0c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.330+0000 7f5b69400640 1 -- 172.21.15.28:0/4062428869 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148474923 0 0) 0x7f5b6c0c64f0 con 0x7f5b6c150d70 2024-09-13T16:44:32.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.330+0000 7f5b69e00640 1 -- 172.21.15.28:0/4062428869 >> v2:172.21.15.191:3300/0 conn(0x7f5b6c0c4550 msgr2=0x7f5b6c0c6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.330+0000 7f5b69e00640 1 --2- 172.21.15.28:0/4062428869 >> v2:172.21.15.191:3300/0 conn(0x7f5b6c0c4550 0x7f5b6c0c6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.330+0000 7f5b69e00640 1 -- 172.21.15.28:0/4062428869 >> v1:172.21.15.28:6789/0 conn(0x7f5b6c150d70 legacy=0x7f5b6c156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.330+0000 7f5b69e00640 1 -- 172.21.15.28:0/4062428869 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c0c5ea0 con 0x7f5b6c150990 2024-09-13T16:44:32.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.331+0000 7f5b69e00640 1 --2- 172.21.15.28:0/4062428869 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b6c150990 0x7f5b6c0c17f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c0097a0 tx=0x7f5b5c02a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7cc34797fe410ee server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.331+0000 7f5b69400640 1 -- 172.21.15.28:0/4062428869 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c032440 con 0x7f5b6c150990 2024-09-13T16:44:32.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.331+0000 7f5b69400640 1 -- 172.21.15.28:0/4062428869 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5c0325a0 con 0x7f5b6c150990 2024-09-13T16:44:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.331+0000 7f5b69400640 1 -- 172.21.15.28:0/4062428869 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c032930 con 0x7f5b6c150990 2024-09-13T16:44:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.332+0000 7f5b6b200640 1 -- 172.21.15.28:0/4062428869 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b6c150990 msgr2=0x7f5b6c0c17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.332+0000 7f5b6b200640 1 --2- 172.21.15.28:0/4062428869 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b6c150990 0x7f5b6c0c17f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c0097a0 tx=0x7f5b5c02a230 comp rx=0 tx=0).stop 2024-09-13T16:44:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.332+0000 7f5b6b200640 1 -- 172.21.15.28:0/4062428869 shutdown_connections 2024-09-13T16:44:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.332+0000 7f5b6b200640 1 --2- 172.21.15.28:0/4062428869 >> v2:172.21.15.191:3300/0 conn(0x7f5b6c0c4550 0x7f5b6c0c6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.332+0000 7f5b6b200640 1 --2- 172.21.15.28:0/4062428869 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b6c150990 0x7f5b6c0c17f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.332+0000 7f5b6b200640 1 -- 172.21.15.28:0/4062428869 >> 172.21.15.28:0/4062428869 conn(0x7f5b6c069e80 msgr2=0x7f5b6c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.332+0000 7f5b6b200640 1 -- 172.21.15.28:0/4062428869 shutdown_connections 2024-09-13T16:44:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.332+0000 7f5b6b200640 1 -- 172.21.15.28:0/4062428869 wait complete. 2024-09-13T16:44:32.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.333+0000 7f5b6b200640 1 Processor -- start 2024-09-13T16:44:32.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.334+0000 7f5b6b200640 1 -- start start 2024-09-13T16:44:32.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.334+0000 7f5b6b200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b6c0c4550 0x7f5b6c0c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.334+0000 7f5b6b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b6c150990 0x7f5b6c153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.334+0000 7f5b6b200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b6c150d70 0x7f5b6c156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.334+0000 7f5b6b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c1ec780 con 0x7f5b6c150990 2024-09-13T16:44:32.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.334+0000 7f5b6b200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c1eccb0 con 0x7f5b6c150d70 2024-09-13T16:44:32.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.334+0000 7f5b6b200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c0c9910 con 0x7f5b6c0c4550 2024-09-13T16:44:32.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.335+0000 7f5b61400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b6c150990 0x7f5b6c153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.335+0000 7f5b69e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b6c0c4550 0x7f5b6c0c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.335+0000 7f5b69e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b6c0c4550 0x7f5b6c0c8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:55290/0 (socket says 172.21.15.28:55290) 2024-09-13T16:44:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.335+0000 7f5b69e00640 1 -- 172.21.15.28:0/2551926684 learned_addr learned my addr 172.21.15.28:0/2551926684 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.335+0000 7f5b6a800640 1 --2- 172.21.15.28:0/2551926684 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b6c150d70 0x7f5b6c156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.335+0000 7f5b61400640 1 -- 172.21.15.28:0/2551926684 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b6c0c4550 msgr2=0x7f5b6c0c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.335+0000 7f5b61400640 1 --2- 172.21.15.28:0/2551926684 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b6c0c4550 0x7f5b6c0c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.335+0000 7f5b61400640 1 -- 172.21.15.28:0/2551926684 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b6c150d70 msgr2=0x7f5b6c156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.335+0000 7f5b61400640 1 --2- 172.21.15.28:0/2551926684 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b6c150d70 0x7f5b6c156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.335+0000 7f5b61400640 1 -- 172.21.15.28:0/2551926684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5c009400 con 0x7f5b6c150990 2024-09-13T16:44:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.335+0000 7f5b69e00640 1 --2- 172.21.15.28:0/2551926684 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b6c0c4550 0x7f5b6c0c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.335+0000 7f5b61400640 1 --2- 172.21.15.28:0/2551926684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b6c150990 0x7f5b6c153990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0c1d30 tx=0x7f5b5400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.335+0000 7f5b62a00640 1 -- 172.21.15.28:0/2551926684 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b540074e0 con 0x7f5b6c150990 2024-09-13T16:44:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.335+0000 7f5b62a00640 1 -- 172.21.15.28:0/2551926684 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b54007640 con 0x7f5b6c150990 2024-09-13T16:44:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.336+0000 7f5b6b200640 1 -- 172.21.15.28:0/2551926684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6c0c8390 con 0x7f5b6c150990 2024-09-13T16:44:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.336+0000 7f5b62a00640 1 -- 172.21.15.28:0/2551926684 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54004a20 con 0x7f5b6c150990 2024-09-13T16:44:32.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.336+0000 7f5b6b200640 1 -- 172.21.15.28:0/2551926684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6c0c8a70 con 0x7f5b6c150990 2024-09-13T16:44:32.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.338+0000 7f5b62a00640 1 -- 172.21.15.28:0/2551926684 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5b54004b80 con 0x7f5b6c150990 2024-09-13T16:44:32.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.338+0000 7f5b62a00640 1 --2- 172.21.15.28:0/2551926684 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f5b4806ac00 0x7f5b4806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.339+0000 7f5b62a00640 1 -- 172.21.15.28:0/2551926684 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5b54088fd0 con 0x7f5b6c150990 2024-09-13T16:44:32.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.339+0000 7f5b69e00640 1 --2- 172.21.15.28:0/2551926684 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f5b4806ac00 0x7f5b4806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.339+0000 7f5b73bd4740 1 Processor -- start 2024-09-13T16:44:32.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.339+0000 7f5b73bd4740 1 -- start start 2024-09-13T16:44:32.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.339+0000 7f5b69e00640 1 --2- 172.21.15.28:0/2551926684 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f5b4806ac00 0x7f5b4806d0c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c02a270 tx=0x7f5b5c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.340+0000 7f5b73bd4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430d0c40 0x5630430cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.340+0000 7f5b73bd4740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430ce470 0x56304311ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.340+0000 7f5b73bd4740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430d1020 0x56304311cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.340+0000 7f5b73bd4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563043038ab0 con 0x5630430d0c40 2024-09-13T16:44:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.340+0000 7f5b73bd4740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x563043036240 con 0x5630430ce470 2024-09-13T16:44:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.340+0000 7f5b73bd4740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x563043031b30 con 0x5630430d1020 2024-09-13T16:44:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.340+0000 7f5b69e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430d0c40 0x5630430cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.341+0000 7f5b69e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430d0c40 0x5630430cb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44172/0 (socket says 172.21.15.28:44172) 2024-09-13T16:44:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.341+0000 7f5b69e00640 1 -- 172.21.15.28:0/2489811640 learned_addr learned my addr 172.21.15.28:0/2489811640 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.341+0000 7f5b6a800640 1 --2- 172.21.15.28:0/2489811640 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430d1020 0x56304311cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.341+0000 7f5b61400640 1 --2- 172.21.15.28:0/2489811640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430ce470 0x56304311ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.341+0000 7f5b69e00640 1 -- 172.21.15.28:0/2489811640 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430d1020 msgr2=0x56304311cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.341+0000 7f5b69e00640 1 --2- 172.21.15.28:0/2489811640 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430d1020 0x56304311cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.341+0000 7f5b69e00640 1 -- 172.21.15.28:0/2489811640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430ce470 msgr2=0x56304311ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.341+0000 7f5b69e00640 1 --2- 172.21.15.28:0/2489811640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430ce470 0x56304311ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.341+0000 7f5b69e00640 1 -- 172.21.15.28:0/2489811640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563042ffcbd0 con 0x5630430d0c40 2024-09-13T16:44:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.341+0000 7f5b69e00640 1 --2- 172.21.15.28:0/2489811640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430d0c40 0x5630430cb0c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c009890 tx=0x7f5b5c02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36425c21a48fbdf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.341+0000 7f5b5aa00640 1 -- 172.21.15.28:0/2489811640 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c03c780 con 0x5630430d0c40 2024-09-13T16:44:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.341+0000 7f5b5aa00640 1 -- 172.21.15.28:0/2489811640 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5c04b070 con 0x5630430d0c40 2024-09-13T16:44:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.341+0000 7f5b5aa00640 1 -- 172.21.15.28:0/2489811640 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c03cae0 con 0x5630430d0c40 2024-09-13T16:44:32.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.342+0000 7f5b73bd4740 1 -- 172.21.15.28:0/2489811640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430d0c40 msgr2=0x5630430cb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.342+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/2489811640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430d0c40 0x5630430cb0c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c009890 tx=0x7f5b5c02c000 comp rx=0 tx=0).stop 2024-09-13T16:44:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.342+0000 7f5b73bd4740 1 -- 172.21.15.28:0/2489811640 shutdown_connections 2024-09-13T16:44:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.342+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/2489811640 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430d1020 0x56304311cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.342+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/2489811640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430ce470 0x56304311ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.342+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/2489811640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430d0c40 0x5630430cb0c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.342+0000 7f5b73bd4740 1 -- 172.21.15.28:0/2489811640 >> 172.21.15.28:0/2489811640 conn(0x5630430cf360 msgr2=0x5630430cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.342+0000 7f5b73bd4740 1 -- 172.21.15.28:0/2489811640 shutdown_connections 2024-09-13T16:44:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.342+0000 7f5b73bd4740 1 -- 172.21.15.28:0/2489811640 wait complete. 2024-09-13T16:44:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.343+0000 7f5b73bd4740 1 -- start start 2024-09-13T16:44:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.343+0000 7f5b73bd4740 1 Processor -- start 2024-09-13T16:44:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.344+0000 7f5b73bd4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430ce470 0x563043106270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.344+0000 7f5b73bd4740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430d0c40 0x563043110d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.344+0000 7f5b73bd4740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430d1020 0x563043104140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.344+0000 7f5b73bd4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563042fd3260 con 0x5630430ce470 2024-09-13T16:44:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.344+0000 7f5b73bd4740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x563042fd2f00 con 0x5630430d0c40 2024-09-13T16:44:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.344+0000 7f5b73bd4740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x563042d98f20 con 0x5630430d1020 2024-09-13T16:44:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.344+0000 7f5b73bd4740 10 client.0 _ll_register_callbacks cb 0x56304302b1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:44:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.344+0000 7f5b69e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430ce470 0x563043106270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.344+0000 7f5b69e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430ce470 0x563043106270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44188/0 (socket says 172.21.15.28:44188) 2024-09-13T16:44:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.344+0000 7f5b69e00640 1 -- 172.21.15.28:0/4012937351 learned_addr learned my addr 172.21.15.28:0/4012937351 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.345+0000 7f5b6a800640 1 --2- 172.21.15.28:0/4012937351 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430d1020 0x563043104140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.345+0000 7f5b61400640 1 --2- 172.21.15.28:0/4012937351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430d0c40 0x563043110d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.345+0000 7f5b69e00640 1 -- 172.21.15.28:0/4012937351 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430d1020 msgr2=0x563043104140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.345+0000 7f5b69e00640 1 --2- 172.21.15.28:0/4012937351 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430d1020 0x563043104140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.345+0000 7f5b69e00640 1 -- 172.21.15.28:0/4012937351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430d0c40 msgr2=0x563043110d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.345+0000 7f5b69e00640 1 --2- 172.21.15.28:0/4012937351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430d0c40 0x563043110d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.345+0000 7f5b69e00640 1 -- 172.21.15.28:0/4012937351 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6c0c5ea0 con 0x5630430ce470 2024-09-13T16:44:32.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.345+0000 7f5b69e00640 1 --2- 172.21.15.28:0/4012937351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430ce470 0x563043106270 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c000c00 tx=0x7f5b5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9983769ba5b4861b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.345+0000 7f5b4f400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T16:44:32.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.345+0000 7f5b4f400640 1 -- 172.21.15.28:0/4012937351 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c052680 con 0x5630430ce470 2024-09-13T16:44:32.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.345+0000 7f5b4f400640 1 -- 172.21.15.28:0/4012937351 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5b5c04d070 con 0x5630430ce470 2024-09-13T16:44:32.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.345+0000 7f5b73bd4740 1 -- 172.21.15.28:0/4012937351 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56304305bcd0 con 0x5630430ce470 2024-09-13T16:44:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.346+0000 7f5b4f400640 1 -- 172.21.15.28:0/4012937351 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b5c0527e0 con 0x5630430ce470 2024-09-13T16:44:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.346+0000 7f5b73bd4740 10 client.4680 fetch_fsmap learned FSMap version 17 2024-09-13T16:44:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.346+0000 7f5b73bd4740 1 -- 172.21.15.28:0/4012937351 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5630430301b0 con 0x5630430ce470 2024-09-13T16:44:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.346+0000 7f5b4f400640 1 -- 172.21.15.28:0/4012937351 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56304305bcd0 con 0x5630430ce470 2024-09-13T16:44:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.346+0000 7f5b4f400640 1 -- 172.21.15.28:0/4012937351 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 17) v1 ==== 4252+0+0 (secure 0 0 0) 0x7f5b5c044040 con 0x5630430ce470 2024-09-13T16:44:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.346+0000 7f5b73bd4740 10 client.4680 fetch_fsmap finished waiting for FSMap version 17 2024-09-13T16:44:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.346+0000 7f5b73bd4740 10 client.4680 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-13T16:44:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.346+0000 7f5b73bd4740 20 client.4680 populate_metadata read hostname 'smithi028' 2024-09-13T16:44:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.346+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/4012937351 >> [v2:172.21.15.28:6836/4068563017,v1:172.21.15.28:6837/4068563017] conn(0x5630430fea80 0x5630430fee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.346+0000 7f5b73bd4740 4 client.4680 mds_command: new command op to 4250 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:44:32.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.346+0000 7f5b73bd4740 1 -- 172.21.15.28:0/4012937351 --> [v2:172.21.15.28:6836/4068563017,v1:172.21.15.28:6837/4068563017] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563042d253c0 con 0x5630430fea80 2024-09-13T16:44:32.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.346+0000 7f5b61400640 1 --2- 172.21.15.28:0/4012937351 >> [v2:172.21.15.28:6836/4068563017,v1:172.21.15.28:6837/4068563017] conn(0x5630430fea80 0x5630430fee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.347+0000 7f5b61400640 1 --2- 172.21.15.28:0/4012937351 >> [v2:172.21.15.28:6836/4068563017,v1:172.21.15.28:6837/4068563017] conn(0x5630430fea80 0x5630430fee60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=24a3649833b584d1 server_cookie=b5213d225de71a75 in_seq=0 out_seq=0 2024-09-13T16:44:32.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.347+0000 7f5b4f400640 10 client.4680 ms_handle_connect on v2:172.21.15.28:6836/4068563017 2024-09-13T16:44:32.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.347+0000 7f5b4f400640 1 -- 172.21.15.28:0/4012937351 <== mds.1 v2:172.21.15.28:6836/4068563017 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563042d253c0 con 0x5630430fea80 2024-09-13T16:44:32.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.347+0000 7f5b4f400640 10 client.4680 handle_command_reply: tid=0 2024-09-13T16:44:32.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.347+0000 7f5b73bd4740 1 client.4680 shutdown 2024-09-13T16:44:32.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.348+0000 7f5b73bd4740 1 -- 172.21.15.28:0/4012937351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430ce470 msgr2=0x563043106270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.348+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/4012937351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430ce470 0x563043106270 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c000c00 tx=0x7f5b5c004730 comp rx=0 tx=0).stop 2024-09-13T16:44:32.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.348+0000 7f5b73bd4740 1 -- 172.21.15.28:0/4012937351 shutdown_connections 2024-09-13T16:44:32.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.348+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/4012937351 >> [v2:172.21.15.28:6836/4068563017,v1:172.21.15.28:6837/4068563017] conn(0x5630430fea80 0x5630430fee60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.348+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/4012937351 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430d1020 0x563043104140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.348+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/4012937351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430d0c40 0x563043110d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.348+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/4012937351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430ce470 0x563043106270 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.348+0000 7f5b4f400640 0 client.4680 ms_handle_reset on v2:172.21.15.28:6836/4068563017 2024-09-13T16:44:32.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.348+0000 7f5b73bd4740 1 -- 172.21.15.28:0/4012937351 >> 172.21.15.28:0/4012937351 conn(0x5630430cf360 msgr2=0x563043108300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.348+0000 7f5b73bd4740 1 -- 172.21.15.28:0/4012937351 shutdown_connections 2024-09-13T16:44:32.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.348+0000 7f5b73bd4740 1 -- 172.21.15.28:0/4012937351 wait complete. 2024-09-13T16:44:32.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.348+0000 7f5b73bd4740 20 client.4680 trim_cache size 0 max 16384 2024-09-13T16:44:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.364+0000 7f5b73bd4740 1 Processor -- start 2024-09-13T16:44:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.364+0000 7f5b73bd4740 1 -- start start 2024-09-13T16:44:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.364+0000 7f5b73bd4740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430ce470 0x5630430ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.364+0000 7f5b73bd4740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430d0c40 0x5630430ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.364+0000 7f5b73bd4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430d1020 0x5630430fee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.364+0000 7f5b73bd4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56304311c070 con 0x5630430d1020 2024-09-13T16:44:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.364+0000 7f5b73bd4740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x56304311c8d0 con 0x5630430d0c40 2024-09-13T16:44:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.364+0000 7f5b73bd4740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x5630430c9860 con 0x5630430ce470 2024-09-13T16:44:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.365+0000 7f5b6a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430d1020 0x5630430fee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.365+0000 7f5b6a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430d1020 0x5630430fee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44196/0 (socket says 172.21.15.28:44196) 2024-09-13T16:44:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.365+0000 7f5b6a800640 1 -- 172.21.15.28:0/1587376109 learned_addr learned my addr 172.21.15.28:0/1587376109 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.365+0000 7f5b69e00640 1 --2- 172.21.15.28:0/1587376109 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430ce470 0x5630430ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.365+0000 7f5b61400640 1 --2- 172.21.15.28:0/1587376109 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430d0c40 0x5630430ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.365+0000 7f5b6a800640 1 -- 172.21.15.28:0/1587376109 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430ce470 msgr2=0x5630430ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.365+0000 7f5b6a800640 1 --2- 172.21.15.28:0/1587376109 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430ce470 0x5630430ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.365+0000 7f5b6a800640 1 -- 172.21.15.28:0/1587376109 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430d0c40 msgr2=0x5630430ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.365+0000 7f5b6a800640 1 --2- 172.21.15.28:0/1587376109 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430d0c40 0x5630430ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.365+0000 7f5b6a800640 1 -- 172.21.15.28:0/1587376109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563043065170 con 0x5630430d1020 2024-09-13T16:44:32.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.365+0000 7f5b6a800640 1 --2- 172.21.15.28:0/1587376109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430d1020 0x5630430fee60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0c81c0 tx=0x7f5b64009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec455d8bcb1fea9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.366+0000 7f5b5b400640 1 -- 172.21.15.28:0/1587376109 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6400e040 con 0x5630430d1020 2024-09-13T16:44:32.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.366+0000 7f5b5b400640 1 -- 172.21.15.28:0/1587376109 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6400f040 con 0x5630430d1020 2024-09-13T16:44:32.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.366+0000 7f5b5b400640 1 -- 172.21.15.28:0/1587376109 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6401f430 con 0x5630430d1020 2024-09-13T16:44:32.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.366+0000 7f5b73bd4740 1 -- 172.21.15.28:0/1587376109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430d1020 msgr2=0x5630430fee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.366+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/1587376109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430d1020 0x5630430fee60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0c81c0 tx=0x7f5b64009750 comp rx=0 tx=0).stop 2024-09-13T16:44:32.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.366+0000 7f5b73bd4740 1 -- 172.21.15.28:0/1587376109 shutdown_connections 2024-09-13T16:44:32.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.366+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/1587376109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430d1020 0x5630430fee60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.366+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/1587376109 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430d0c40 0x5630430ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.366+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/1587376109 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430ce470 0x5630430ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.366+0000 7f5b73bd4740 1 -- 172.21.15.28:0/1587376109 >> 172.21.15.28:0/1587376109 conn(0x5630430cf360 msgr2=0x5630431102c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.366+0000 7f5b73bd4740 1 -- 172.21.15.28:0/1587376109 shutdown_connections 2024-09-13T16:44:32.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.366+0000 7f5b73bd4740 1 -- 172.21.15.28:0/1587376109 wait complete. 2024-09-13T16:44:32.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.367+0000 7f5b73bd4740 1 -- start start 2024-09-13T16:44:32.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.367+0000 7f5b73bd4740 1 Processor -- start 2024-09-13T16:44:32.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.368+0000 7f5b73bd4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430ce470 0x5630431789d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.368+0000 7f5b73bd4740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430d0c40 0x5630431079e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.368+0000 7f5b73bd4740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430d1020 0x563043107f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.368+0000 7f5b73bd4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563043111240 con 0x5630430ce470 2024-09-13T16:44:32.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.368+0000 7f5b73bd4740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630431087c0 con 0x5630430d0c40 2024-09-13T16:44:32.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.368+0000 7f5b73bd4740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x563043182550 con 0x5630430d1020 2024-09-13T16:44:32.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.368+0000 7f5b73bd4740 10 client.0 _ll_register_callbacks cb 0x7f5b54007f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:44:32.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.368+0000 7f5b69e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430ce470 0x5630431789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.368+0000 7f5b69e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430ce470 0x5630431789d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44206/0 (socket says 172.21.15.28:44206) 2024-09-13T16:44:32.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.368+0000 7f5b6a800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430d1020 0x563043107f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.368+0000 7f5b61400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430d0c40 0x5630431079e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.368+0000 7f5b69e00640 1 -- 172.21.15.28:0/1614931867 learned_addr learned my addr 172.21.15.28:0/1614931867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:32.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.368+0000 7f5b69e00640 1 -- 172.21.15.28:0/1614931867 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430d1020 msgr2=0x563043107f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.368+0000 7f5b69e00640 1 --2- 172.21.15.28:0/1614931867 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430d1020 0x563043107f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.368+0000 7f5b69e00640 1 -- 172.21.15.28:0/1614931867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430d0c40 msgr2=0x5630431079e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.368+0000 7f5b69e00640 1 --2- 172.21.15.28:0/1614931867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430d0c40 0x5630431079e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.368+0000 7f5b69e00640 1 -- 172.21.15.28:0/1614931867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b64017410 con 0x5630430ce470 2024-09-13T16:44:32.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.369+0000 7f5b69e00640 1 --2- 172.21.15.28:0/1614931867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430ce470 0x5630431789d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c047de0 tx=0x7f5b5c008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ddd0333da8f2e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.369+0000 7f5b4f400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T16:44:32.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.369+0000 7f5b4f400640 1 -- 172.21.15.28:0/1614931867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c052890 con 0x5630430ce470 2024-09-13T16:44:32.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.369+0000 7f5b4f400640 1 -- 172.21.15.28:0/1614931867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5b5c04d030 con 0x5630430ce470 2024-09-13T16:44:32.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.369+0000 7f5b73bd4740 1 -- 172.21.15.28:0/1614931867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5b5c04b070 con 0x5630430ce470 2024-09-13T16:44:32.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.369+0000 7f5b4f400640 1 -- 172.21.15.28:0/1614931867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b5c047e90 con 0x5630430ce470 2024-09-13T16:44:32.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.369+0000 7f5b73bd4740 10 client.4686 fetch_fsmap learned FSMap version 17 2024-09-13T16:44:32.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.369+0000 7f5b73bd4740 1 -- 172.21.15.28:0/1614931867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x563042fd2470 con 0x5630430ce470 2024-09-13T16:44:32.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.370+0000 7f5b4f400640 1 -- 172.21.15.28:0/1614931867 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b5c04b070 con 0x5630430ce470 2024-09-13T16:44:32.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.370+0000 7f5b4f400640 1 -- 172.21.15.28:0/1614931867 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 17) v1 ==== 4252+0+0 (secure 0 0 0) 0x7f5b5c044070 con 0x5630430ce470 2024-09-13T16:44:32.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.370+0000 7f5b73bd4740 10 client.4686 fetch_fsmap finished waiting for FSMap version 17 2024-09-13T16:44:32.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.370+0000 7f5b73bd4740 10 client.4686 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-13T16:44:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.370+0000 7f5b73bd4740 20 client.4686 populate_metadata read hostname 'smithi028' 2024-09-13T16:44:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.370+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/1614931867 >> [v2:172.21.15.28:6836/4068563017,v1:172.21.15.28:6837/4068563017] conn(0x56304312a1e0 0x56304317e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.370+0000 7f5b73bd4740 4 client.4686 mds_command: new command op to 4250 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:44:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.370+0000 7f5b73bd4740 1 -- 172.21.15.28:0/1614931867 --> [v2:172.21.15.28:6836/4068563017,v1:172.21.15.28:6837/4068563017] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563042d366c0 con 0x56304312a1e0 2024-09-13T16:44:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.370+0000 7f5b61400640 1 --2- 172.21.15.28:0/1614931867 >> [v2:172.21.15.28:6836/4068563017,v1:172.21.15.28:6837/4068563017] conn(0x56304312a1e0 0x56304317e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.370+0000 7f5b61400640 1 --2- 172.21.15.28:0/1614931867 >> [v2:172.21.15.28:6836/4068563017,v1:172.21.15.28:6837/4068563017] conn(0x56304312a1e0 0x56304317e8e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5259ce57428dc6dd server_cookie=cda50b96e6a70214 in_seq=0 out_seq=0 2024-09-13T16:44:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.370+0000 7f5b4f400640 10 client.4686 ms_handle_connect on v2:172.21.15.28:6836/4068563017 2024-09-13T16:44:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.371+0000 7f5b4f400640 1 -- 172.21.15.28:0/1614931867 <== mds.1 v2:172.21.15.28:6836/4068563017 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x563042d366c0 con 0x56304312a1e0 2024-09-13T16:44:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.371+0000 7f5b4f400640 10 client.4686 handle_command_reply: tid=0 2024-09-13T16:44:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.371+0000 7f5b73bd4740 1 client.4686 shutdown 2024-09-13T16:44:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.371+0000 7f5b73bd4740 1 -- 172.21.15.28:0/1614931867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430ce470 msgr2=0x5630431789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.371+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/1614931867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430ce470 0x5630431789d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c047de0 tx=0x7f5b5c008030 comp rx=0 tx=0).stop 2024-09-13T16:44:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.371+0000 7f5b73bd4740 1 -- 172.21.15.28:0/1614931867 shutdown_connections 2024-09-13T16:44:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.371+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/1614931867 >> [v2:172.21.15.28:6836/4068563017,v1:172.21.15.28:6837/4068563017] conn(0x56304312a1e0 0x56304317e8e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.371+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/1614931867 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x5630430d1020 0x563043107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.371+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/1614931867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5630430d0c40 0x5630431079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.371+0000 7f5b73bd4740 1 --2- 172.21.15.28:0/1614931867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630430ce470 0x5630431789d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.371+0000 7f5b4f400640 0 client.4686 ms_handle_reset on v2:172.21.15.28:6836/4068563017 2024-09-13T16:44:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.371+0000 7f5b73bd4740 1 -- 172.21.15.28:0/1614931867 >> 172.21.15.28:0/1614931867 conn(0x5630430cf360 msgr2=0x5630430ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.371+0000 7f5b73bd4740 1 -- 172.21.15.28:0/1614931867 shutdown_connections 2024-09-13T16:44:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.371+0000 7f5b73bd4740 1 -- 172.21.15.28:0/1614931867 wait complete. 2024-09-13T16:44:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.371+0000 7f5b73bd4740 20 client.4686 trim_cache size 0 max 16384 2024-09-13T16:44:32.377 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:44:32.378 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:44:32.378 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-09-13T16:44:32.378 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4250, 2024-09-13T16:44:32.378 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T16:44:32.378 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T16:44:32.378 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T16:44:32.378 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 28.360229568000001, 2024-09-13T16:44:32.378 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 17, 2024-09-13T16:44:32.378 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 20, 2024-09-13T16:44:32.378 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 20, 2024-09-13T16:44:32.378 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 42.091219553999998 2024-09-13T16:44:32.378 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:44:32.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.372+0000 7f5b6b200640 1 -- 172.21.15.28:0/2551926684 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f5b4806ac00 msgr2=0x7f5b4806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.372+0000 7f5b6b200640 1 --2- 172.21.15.28:0/2551926684 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f5b4806ac00 0x7f5b4806d0c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c02a270 tx=0x7f5b5c02e000 comp rx=0 tx=0).stop 2024-09-13T16:44:32.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.372+0000 7f5b6b200640 1 -- 172.21.15.28:0/2551926684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b6c150990 msgr2=0x7f5b6c153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.372+0000 7f5b6b200640 1 --2- 172.21.15.28:0/2551926684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b6c150990 0x7f5b6c153990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0c1d30 tx=0x7f5b5400c8e0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.372+0000 7f5b6b200640 1 -- 172.21.15.28:0/2551926684 shutdown_connections 2024-09-13T16:44:32.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.372+0000 7f5b6b200640 1 --2- 172.21.15.28:0/2551926684 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f5b4806ac00 0x7f5b4806d0c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.372+0000 7f5b6b200640 1 --2- 172.21.15.28:0/2551926684 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5b6c150d70 0x7f5b6c156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.372+0000 7f5b6b200640 1 --2- 172.21.15.28:0/2551926684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b6c150990 0x7f5b6c153990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.372+0000 7f5b6b200640 1 --2- 172.21.15.28:0/2551926684 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5b6c0c4550 0x7f5b6c0c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.372+0000 7f5b6b200640 1 -- 172.21.15.28:0/2551926684 >> 172.21.15.28:0/2551926684 conn(0x7f5b6c069e80 msgr2=0x7f5b6c14b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.372+0000 7f5b6b200640 1 -- 172.21.15.28:0/2551926684 shutdown_connections 2024-09-13T16:44:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.372+0000 7f5b6b200640 1 -- 172.21.15.28:0/2551926684 wait complete. 2024-09-13T16:44:32.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-13T16:44:32.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.543+0000 7ffbbf400640 1 Processor -- start 2024-09-13T16:44:32.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.543+0000 7ffbbf400640 1 -- start start 2024-09-13T16:44:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.544+0000 7ffbbf400640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ffbb8150ca0 0x7ffbb81568e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.544+0000 7ffbbf400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb8151080 con 0x7ffbb80c4d60 2024-09-13T16:44:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.544+0000 7ffbbf400640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb806a990 con 0x7ffbb80c2250 2024-09-13T16:44:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.544+0000 7ffbbf400640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbb80c2670 con 0x7ffbb8150ca0 2024-09-13T16:44:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.544+0000 7ffbbd400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ffbb80c4d60 0x7ffbb8161220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53774/0 (socket says 172.21.15.28:53774) 2024-09-13T16:44:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.544+0000 7ffbbd400640 1 -- 172.21.15.28:0/2965241703 learned_addr learned my addr 172.21.15.28:0/2965241703 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.545+0000 7ffbbca00640 1 --2- 172.21.15.28:0/2965241703 >> v2:172.21.15.191:3300/0 conn(0x7ffbb8150ca0 0x7ffbb81568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.545+0000 7ffbb7400640 1 -- 172.21.15.28:0/2965241703 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054850898 0 0) 0x7ffbb8151080 con 0x7ffbb80c4d60 2024-09-13T16:44:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.545+0000 7ffbb7400640 1 -- 172.21.15.28:0/2965241703 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba0003520 con 0x7ffbb80c4d60 2024-09-13T16:44:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.545+0000 7ffbb7400640 1 -- 172.21.15.28:0/2965241703 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801842876 0 0) 0x7ffbb806a990 con 0x7ffbb80c2250 2024-09-13T16:44:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.545+0000 7ffbb7400640 1 -- 172.21.15.28:0/2965241703 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbb8151080 con 0x7ffbb80c2250 2024-09-13T16:44:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.545+0000 7ffbbca00640 1 -- 172.21.15.28:0/2965241703 >> v1:172.21.15.191:6789/0 conn(0x7ffbb80c2250 legacy=0x7ffbb815a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.545+0000 7ffbbca00640 1 -- 172.21.15.28:0/2965241703 >> v1:172.21.15.28:6789/0 conn(0x7ffbb80c4d60 legacy=0x7ffbb8161220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.545+0000 7ffbb7400640 1 -- 172.21.15.28:0/2965241703 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1899576860 0 0) 0x7ffba0003520 con 0x7ffbb80c4d60 2024-09-13T16:44:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.545+0000 7ffbbca00640 1 -- 172.21.15.28:0/2965241703 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb80c2870 con 0x7ffbb8150ca0 2024-09-13T16:44:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.546+0000 7ffbbca00640 1 --2- 172.21.15.28:0/2965241703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbb8150ca0 0x7ffbb81568e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0097a0 tx=0x7ffbac02a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ef53f9df274ac44 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.546+0000 7ffbb7400640 1 -- 172.21.15.28:0/2965241703 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00019f0 con 0x7ffbb8150ca0 2024-09-13T16:44:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.546+0000 7ffbb7400640 1 -- 172.21.15.28:0/2965241703 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbac02abc0 con 0x7ffbb8150ca0 2024-09-13T16:44:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.546+0000 7ffbb7400640 1 -- 172.21.15.28:0/2965241703 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac032440 con 0x7ffbb8150ca0 2024-09-13T16:44:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.547+0000 7ffbbf400640 1 -- 172.21.15.28:0/2965241703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbb8150ca0 msgr2=0x7ffbb81568e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.547+0000 7ffbbf400640 1 --2- 172.21.15.28:0/2965241703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbb8150ca0 0x7ffbb81568e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0097a0 tx=0x7ffbac02a130 comp rx=0 tx=0).stop 2024-09-13T16:44:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.547+0000 7ffbbf400640 1 -- 172.21.15.28:0/2965241703 shutdown_connections 2024-09-13T16:44:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.547+0000 7ffbbf400640 1 --2- 172.21.15.28:0/2965241703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbb8150ca0 0x7ffbb81568e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.547+0000 7ffbbf400640 1 -- 172.21.15.28:0/2965241703 >> 172.21.15.28:0/2965241703 conn(0x7ffbb8066040 msgr2=0x7ffbb8069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.547+0000 7ffbbf400640 1 -- 172.21.15.28:0/2965241703 shutdown_connections 2024-09-13T16:44:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.547+0000 7ffbbf400640 1 -- 172.21.15.28:0/2965241703 wait complete. 2024-09-13T16:44:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.549+0000 7ffbbf400640 1 Processor -- start 2024-09-13T16:44:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.549+0000 7ffbbf400640 1 -- start start 2024-09-13T16:44:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.550+0000 7ffbbf400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbb80c2250 0x7ffbb814c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.550+0000 7ffbbf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb80c4d60 0x7ffbb8147fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.550+0000 7ffbbf400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbb8150ca0 0x7ffbb80bfa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.550+0000 7ffbbf400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb80bf5c0 con 0x7ffbb80c4d60 2024-09-13T16:44:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.550+0000 7ffbbf400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb81489c0 con 0x7ffbb8150ca0 2024-09-13T16:44:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.550+0000 7ffbbf400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb8148ca0 con 0x7ffbb80c2250 2024-09-13T16:44:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.550+0000 7ffbb7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb80c4d60 0x7ffbb8147fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.550+0000 7ffbb7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb80c4d60 0x7ffbb8147fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44220/0 (socket says 172.21.15.28:44220) 2024-09-13T16:44:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.550+0000 7ffbb7e00640 1 -- 172.21.15.28:0/246204845 learned_addr learned my addr 172.21.15.28:0/246204845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.550+0000 7ffbbca00640 1 --2- 172.21.15.28:0/246204845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbb80c2250 0x7ffbb814c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.550+0000 7ffbbd400640 1 --2- 172.21.15.28:0/246204845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbb8150ca0 0x7ffbb80bfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.550+0000 7ffbb7e00640 1 -- 172.21.15.28:0/246204845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbb80c2250 msgr2=0x7ffbb814c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.550+0000 7ffbb7e00640 1 --2- 172.21.15.28:0/246204845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbb80c2250 0x7ffbb814c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.550+0000 7ffbb7e00640 1 -- 172.21.15.28:0/246204845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbb8150ca0 msgr2=0x7ffbb80bfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.550+0000 7ffbb7e00640 1 --2- 172.21.15.28:0/246204845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbb8150ca0 0x7ffbb80bfa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.550+0000 7ffbb7e00640 1 -- 172.21.15.28:0/246204845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbac009400 con 0x7ffbb80c4d60 2024-09-13T16:44:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.550+0000 7ffbbd400640 1 --2- 172.21.15.28:0/246204845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbb8150ca0 0x7ffbb80bfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.551+0000 7ffbb7e00640 1 --2- 172.21.15.28:0/246204845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb80c4d60 0x7ffbb8147fe0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffba400c490 tx=0x7ffba400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.551+0000 7ffbb4c00640 1 -- 172.21.15.28:0/246204845 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba40052d0 con 0x7ffbb80c4d60 2024-09-13T16:44:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.551+0000 7ffbb4c00640 1 -- 172.21.15.28:0/246204845 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba4005430 con 0x7ffbb80c4d60 2024-09-13T16:44:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.551+0000 7ffbb4c00640 1 -- 172.21.15.28:0/246204845 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba4012870 con 0x7ffbb80c4d60 2024-09-13T16:44:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.551+0000 7ffbbf400640 1 -- 172.21.15.28:0/246204845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb81eca60 con 0x7ffbb80c4d60 2024-09-13T16:44:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.551+0000 7ffbbf400640 1 -- 172.21.15.28:0/246204845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb80c1050 con 0x7ffbb80c4d60 2024-09-13T16:44:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.553+0000 7ffbb4c00640 1 -- 172.21.15.28:0/246204845 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ffba40129d0 con 0x7ffbb80c4d60 2024-09-13T16:44:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.554+0000 7ffbb4c00640 1 --2- 172.21.15.28:0/246204845 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ffb9806ac30 0x7ffb9806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.554+0000 7ffbb4c00640 1 -- 172.21.15.28:0/246204845 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffba4088fb0 con 0x7ffbb80c4d60 2024-09-13T16:44:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.554+0000 7ffbbca00640 1 --2- 172.21.15.28:0/246204845 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ffb9806ac30 0x7ffb9806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.555+0000 7ffbc2218740 1 Processor -- start 2024-09-13T16:44:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.555+0000 7ffbc2218740 1 -- start start 2024-09-13T16:44:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.555+0000 7ffbbca00640 1 --2- 172.21.15.28:0/246204845 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ffb9806ac30 0x7ffb9806d0f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ffbac034000 tx=0x7ffbac02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.555+0000 7ffbc2218740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15542ad0 0x555f1553cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.556+0000 7ffbc2218740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 0x555f1558e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.556+0000 7ffbc2218740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f1558ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.556+0000 7ffbc2218740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f154aaab0 con 0x555f15542eb0 2024-09-13T16:44:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.556+0000 7ffbc2218740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f154a8240 con 0x555f15540300 2024-09-13T16:44:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.556+0000 7ffbc2218740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f154a3b30 con 0x555f15542ad0 2024-09-13T16:44:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.556+0000 7ffbbca00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15542ad0 0x555f1553cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.556+0000 7ffbb7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 0x555f1558e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.556+0000 7ffbbd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f1558ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.556+0000 7ffbbca00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15542ad0 0x555f1553cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:55346/0 (socket says 172.21.15.28:55346) 2024-09-13T16:44:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.556+0000 7ffbb7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 0x555f1558e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:50762/0 (socket says 172.21.15.28:50762) 2024-09-13T16:44:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.556+0000 7ffbbca00640 1 -- 172.21.15.28:0/3021468108 learned_addr learned my addr 172.21.15.28:0/3021468108 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.556+0000 7ffbb7e00640 1 -- 172.21.15.28:0/3021468108 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15542ad0 msgr2=0x555f1553cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.556+0000 7ffbb7e00640 1 --2- 172.21.15.28:0/3021468108 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15542ad0 0x555f1553cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.556+0000 7ffbb7e00640 1 -- 172.21.15.28:0/3021468108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 msgr2=0x555f1558ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.556+0000 7ffbb7e00640 1 --2- 172.21.15.28:0/3021468108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f1558ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.556+0000 7ffbb7e00640 1 -- 172.21.15.28:0/3021468108 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f1546ebd0 con 0x555f15540300 2024-09-13T16:44:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.556+0000 7ffbb7e00640 1 --2- 172.21.15.28:0/3021468108 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 0x555f1558e900 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffba4087440 tx=0x7ffba4000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22e4c2bf25c8af4d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.557+0000 7ffba9600640 1 -- 172.21.15.28:0/3021468108 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba40560c0 con 0x555f15540300 2024-09-13T16:44:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.557+0000 7ffba9600640 1 -- 172.21.15.28:0/3021468108 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba4054430 con 0x555f15540300 2024-09-13T16:44:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.557+0000 7ffba9600640 1 -- 172.21.15.28:0/3021468108 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba4012e80 con 0x555f15540300 2024-09-13T16:44:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.557+0000 7ffbc2218740 1 -- 172.21.15.28:0/3021468108 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 msgr2=0x555f1558e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.557+0000 7ffbc2218740 1 --2- 172.21.15.28:0/3021468108 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 0x555f1558e900 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffba4087440 tx=0x7ffba4000ed0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.557+0000 7ffbc2218740 1 -- 172.21.15.28:0/3021468108 shutdown_connections 2024-09-13T16:44:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.557+0000 7ffbc2218740 1 --2- 172.21.15.28:0/3021468108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f1558ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.557+0000 7ffbc2218740 1 --2- 172.21.15.28:0/3021468108 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 0x555f1558e900 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.557+0000 7ffbc2218740 1 --2- 172.21.15.28:0/3021468108 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15542ad0 0x555f1553cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.557+0000 7ffbc2218740 1 -- 172.21.15.28:0/3021468108 >> 172.21.15.28:0/3021468108 conn(0x555f155411f0 msgr2=0x555f155415f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.557+0000 7ffbc2218740 1 -- 172.21.15.28:0/3021468108 shutdown_connections 2024-09-13T16:44:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.557+0000 7ffbc2218740 1 -- 172.21.15.28:0/3021468108 wait complete. 2024-09-13T16:44:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.558+0000 7ffbc2218740 1 -- start start 2024-09-13T16:44:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.558+0000 7ffbc2218740 1 Processor -- start 2024-09-13T16:44:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.559+0000 7ffbc2218740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 0x555f15584700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.559+0000 7ffbc2218740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15542ad0 0x555f155995b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.559+0000 7ffbc2218740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f15578cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.559+0000 7ffbc2218740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f15445260 con 0x555f15542eb0 2024-09-13T16:44:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.559+0000 7ffbc2218740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f15444f00 con 0x555f15540300 2024-09-13T16:44:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.559+0000 7ffbc2218740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f1520af20 con 0x555f15542ad0 2024-09-13T16:44:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.559+0000 7ffbc2218740 10 client.0 _ll_register_callbacks cb 0x555f1549d1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:44:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.559+0000 7ffbbd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f15578cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.559+0000 7ffbbd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f15578cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44246/0 (socket says 172.21.15.28:44246) 2024-09-13T16:44:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.559+0000 7ffbbd400640 1 -- 172.21.15.28:0/1436415180 learned_addr learned my addr 172.21.15.28:0/1436415180 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.559+0000 7ffbbca00640 1 --2- 172.21.15.28:0/1436415180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 0x555f15584700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.559+0000 7ffbb7e00640 1 --2- 172.21.15.28:0/1436415180 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15542ad0 0x555f155995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.560+0000 7ffbbd400640 1 -- 172.21.15.28:0/1436415180 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15542ad0 msgr2=0x555f155995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.560+0000 7ffbbd400640 1 --2- 172.21.15.28:0/1436415180 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15542ad0 0x555f155995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.560+0000 7ffbbd400640 1 -- 172.21.15.28:0/1436415180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 msgr2=0x555f15584700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.560+0000 7ffbbd400640 1 --2- 172.21.15.28:0/1436415180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 0x555f15584700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.560+0000 7ffbbd400640 1 -- 172.21.15.28:0/1436415180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb80c1050 con 0x555f15542eb0 2024-09-13T16:44:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.560+0000 7ffbb7e00640 1 --2- 172.21.15.28:0/1436415180 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15542ad0 0x555f155995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.560+0000 7ffbbd400640 1 --2- 172.21.15.28:0/1436415180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f15578cd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0011d60 tx=0x7ffbb000f5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f755bc0dde0755e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.560+0000 7ffb9e000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T16:44:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.560+0000 7ffb9e000640 1 -- 172.21.15.28:0/1436415180 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb000fcb0 con 0x555f15542eb0 2024-09-13T16:44:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.560+0000 7ffb9e000640 1 -- 172.21.15.28:0/1436415180 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffbb000fe10 con 0x555f15542eb0 2024-09-13T16:44:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.560+0000 7ffbc2218740 1 -- 172.21.15.28:0/1436415180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555f154cdcd0 con 0x555f15542eb0 2024-09-13T16:44:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.560+0000 7ffb9e000640 1 -- 172.21.15.28:0/1436415180 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffbb0015470 con 0x555f15542eb0 2024-09-13T16:44:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.560+0000 7ffbc2218740 10 client.4698 fetch_fsmap learned FSMap version 17 2024-09-13T16:44:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.560+0000 7ffbc2218740 1 -- 172.21.15.28:0/1436415180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x555f154a21b0 con 0x555f15542eb0 2024-09-13T16:44:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.561+0000 7ffb9e000640 1 -- 172.21.15.28:0/1436415180 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555f154cdcd0 con 0x555f15542eb0 2024-09-13T16:44:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.561+0000 7ffb9e000640 1 -- 172.21.15.28:0/1436415180 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 17) v1 ==== 4252+0+0 (secure 0 0 0) 0x7ffbb00155d0 con 0x555f15542eb0 2024-09-13T16:44:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.561+0000 7ffbc2218740 10 client.4698 fetch_fsmap finished waiting for FSMap version 17 2024-09-13T16:44:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.561+0000 7ffbc2218740 10 client.4698 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T16:44:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.561+0000 7ffbc2218740 20 client.4698 populate_metadata read hostname 'smithi028' 2024-09-13T16:44:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.561+0000 7ffbc2218740 1 --2- 172.21.15.28:0/1436415180 >> [v2:172.21.15.191:6834/704943585,v1:172.21.15.191:6835/704943585] conn(0x555f155799f0 0x555f15577f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.561+0000 7ffbc2218740 4 client.4698 mds_command: new command op to 4258 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:44:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.561+0000 7ffbc2218740 1 -- 172.21.15.28:0/1436415180 --> [v2:172.21.15.191:6834/704943585,v1:172.21.15.191:6835/704943585] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555f151973c0 con 0x555f155799f0 2024-09-13T16:44:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.561+0000 7ffbbca00640 1 --2- 172.21.15.28:0/1436415180 >> [v2:172.21.15.191:6834/704943585,v1:172.21.15.191:6835/704943585] conn(0x555f155799f0 0x555f15577f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.562+0000 7ffbbca00640 1 --2- 172.21.15.28:0/1436415180 >> [v2:172.21.15.191:6834/704943585,v1:172.21.15.191:6835/704943585] conn(0x555f155799f0 0x555f15577f70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a22b7ed3229cabe server_cookie=c5aa79825f7800d2 in_seq=0 out_seq=0 2024-09-13T16:44:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.562+0000 7ffb9e000640 10 client.4698 ms_handle_connect on v2:172.21.15.191:6834/704943585 2024-09-13T16:44:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.562+0000 7ffb9e000640 1 -- 172.21.15.28:0/1436415180 <== mds.0 v2:172.21.15.191:6834/704943585 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555f151973c0 con 0x555f155799f0 2024-09-13T16:44:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.562+0000 7ffb9e000640 10 client.4698 handle_command_reply: tid=0 2024-09-13T16:44:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.562+0000 7ffbc2218740 1 client.4698 shutdown 2024-09-13T16:44:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.563+0000 7ffbc2218740 1 -- 172.21.15.28:0/1436415180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 msgr2=0x555f15578cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.563+0000 7ffbc2218740 1 --2- 172.21.15.28:0/1436415180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f15578cd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0011d60 tx=0x7ffbb000f5a0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.563+0000 7ffbc2218740 1 -- 172.21.15.28:0/1436415180 shutdown_connections 2024-09-13T16:44:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.563+0000 7ffbc2218740 1 --2- 172.21.15.28:0/1436415180 >> [v2:172.21.15.191:6834/704943585,v1:172.21.15.191:6835/704943585] conn(0x555f155799f0 0x555f15577f70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.563+0000 7ffbc2218740 1 --2- 172.21.15.28:0/1436415180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f15578cd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.563+0000 7ffbc2218740 1 --2- 172.21.15.28:0/1436415180 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15542ad0 0x555f155995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.563+0000 7ffb9e000640 0 client.4698 ms_handle_reset on v2:172.21.15.191:6834/704943585 2024-09-13T16:44:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.563+0000 7ffbc2218740 1 --2- 172.21.15.28:0/1436415180 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 0x555f15584700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.563+0000 7ffbc2218740 1 -- 172.21.15.28:0/1436415180 >> 172.21.15.28:0/1436415180 conn(0x555f155411f0 msgr2=0x555f1557a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.563+0000 7ffbc2218740 1 -- 172.21.15.28:0/1436415180 shutdown_connections 2024-09-13T16:44:32.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.563+0000 7ffbc2218740 1 -- 172.21.15.28:0/1436415180 wait complete. 2024-09-13T16:44:32.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.563+0000 7ffbc2218740 20 client.4698 trim_cache size 0 max 16384 2024-09-13T16:44:32.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.578+0000 7ffbc2218740 1 Processor -- start 2024-09-13T16:44:32.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.578+0000 7ffbc2218740 1 -- start start 2024-09-13T16:44:32.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.579+0000 7ffbc2218740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 0x555f1556fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.579+0000 7ffbc2218740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15542ad0 0x555f15570410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.579+0000 7ffbc2218740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f1557a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.579+0000 7ffbc2218740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f1553de40 con 0x555f15542eb0 2024-09-13T16:44:32.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.579+0000 7ffbc2218740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f155711a0 con 0x555f15540300 2024-09-13T16:44:32.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.579+0000 7ffbc2218740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f1558dd00 con 0x555f15542ad0 2024-09-13T16:44:32.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.579+0000 7ffbbd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f1557a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.580+0000 7ffbbd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f1557a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44254/0 (socket says 172.21.15.28:44254) 2024-09-13T16:44:32.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.580+0000 7ffbbd400640 1 -- 172.21.15.28:0/2960362053 learned_addr learned my addr 172.21.15.28:0/2960362053 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:32.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.580+0000 7ffbb7e00640 1 --2- 172.21.15.28:0/2960362053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15542ad0 0x555f15570410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.580+0000 7ffbbca00640 1 --2- 172.21.15.28:0/2960362053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 0x555f1556fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.580+0000 7ffbbd400640 1 -- 172.21.15.28:0/2960362053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15542ad0 msgr2=0x555f15570410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.580+0000 7ffbbd400640 1 --2- 172.21.15.28:0/2960362053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15542ad0 0x555f15570410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.580+0000 7ffbbd400640 1 -- 172.21.15.28:0/2960362053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 msgr2=0x555f1556fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.580+0000 7ffbbd400640 1 --2- 172.21.15.28:0/2960362053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 0x555f1556fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.580+0000 7ffbbd400640 1 -- 172.21.15.28:0/2960362053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f154d7170 con 0x555f15542eb0 2024-09-13T16:44:32.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.580+0000 7ffbbca00640 1 --2- 172.21.15.28:0/2960362053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 0x555f1556fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:32.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.580+0000 7ffbbd400640 1 --2- 172.21.15.28:0/2960362053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f1557a170 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0002230 tx=0x7ffbb000c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc08a2260595eb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.580+0000 7ffbaa000640 1 -- 172.21.15.28:0/2960362053 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00110c0 con 0x555f15542eb0 2024-09-13T16:44:32.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.580+0000 7ffbaa000640 1 -- 172.21.15.28:0/2960362053 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb000faa0 con 0x555f15542eb0 2024-09-13T16:44:32.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.580+0000 7ffbaa000640 1 -- 172.21.15.28:0/2960362053 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0015670 con 0x555f15542eb0 2024-09-13T16:44:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.581+0000 7ffbc2218740 1 -- 172.21.15.28:0/2960362053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 msgr2=0x555f1557a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.581+0000 7ffbc2218740 1 --2- 172.21.15.28:0/2960362053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f1557a170 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0002230 tx=0x7ffbb000c000 comp rx=0 tx=0).stop 2024-09-13T16:44:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.581+0000 7ffbc2218740 1 -- 172.21.15.28:0/2960362053 shutdown_connections 2024-09-13T16:44:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.581+0000 7ffbc2218740 1 --2- 172.21.15.28:0/2960362053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f1557a170 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.581+0000 7ffbc2218740 1 --2- 172.21.15.28:0/2960362053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15542ad0 0x555f15570410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.581+0000 7ffbc2218740 1 --2- 172.21.15.28:0/2960362053 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15540300 0x555f1556fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.581+0000 7ffbc2218740 1 -- 172.21.15.28:0/2960362053 >> 172.21.15.28:0/2960362053 conn(0x555f155411f0 msgr2=0x555f155415d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.581+0000 7ffbc2218740 1 -- 172.21.15.28:0/2960362053 shutdown_connections 2024-09-13T16:44:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.581+0000 7ffbc2218740 1 -- 172.21.15.28:0/2960362053 wait complete. 2024-09-13T16:44:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.582+0000 7ffbc2218740 1 -- start start 2024-09-13T16:44:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.582+0000 7ffbc2218740 1 Processor -- start 2024-09-13T16:44:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbc2218740 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15540300 0x555f155f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbc2218740 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15542ad0 0x555f1557f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbc2218740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f155765a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbc2218740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f15575910 con 0x555f15542eb0 2024-09-13T16:44:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbc2218740 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f155f4450 con 0x555f15542ad0 2024-09-13T16:44:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbc2218740 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f15585820 con 0x555f15540300 2024-09-13T16:44:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbc2218740 10 client.0 _ll_register_callbacks cb 0x7ffbb0001f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:44:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbbca00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15540300 0x555f155f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbbd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f155765a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbbca00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15540300 0x555f155f3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:55376/0 (socket says 172.21.15.28:55376) 2024-09-13T16:44:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbbca00640 1 -- 172.21.15.28:0/3444991272 learned_addr learned my addr 172.21.15.28:0/3444991272 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbb7e00640 1 --2- 172.21.15.28:0/3444991272 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15542ad0 0x555f1557f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbbca00640 1 -- 172.21.15.28:0/3444991272 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15542ad0 msgr2=0x555f1557f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbbca00640 1 --2- 172.21.15.28:0/3444991272 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15542ad0 0x555f1557f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbbca00640 1 -- 172.21.15.28:0/3444991272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 msgr2=0x555f155765a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbbca00640 1 --2- 172.21.15.28:0/3444991272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f155765a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbbca00640 1 -- 172.21.15.28:0/3444991272 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555f154a21b0 con 0x555f15540300 2024-09-13T16:44:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.583+0000 7ffbbca00640 1 --2- 172.21.15.28:0/3444991272 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15540300 0x555f155f3a80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0098c0 tx=0x7ffbac02c070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77a47ed8c27778b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.584+0000 7ffb9e000640 10 client.0 ms_handle_connect on v2:172.21.15.191:3301/0 2024-09-13T16:44:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.584+0000 7ffb9e000640 1 -- 172.21.15.28:0/3444991272 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac04e040 con 0x555f15540300 2024-09-13T16:44:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.584+0000 7ffb9e000640 1 -- 172.21.15.28:0/3444991272 <== mon.2 v2:172.21.15.191:3301/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffbac051740 con 0x555f15540300 2024-09-13T16:44:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.584+0000 7ffbc2218740 1 -- 172.21.15.28:0/3444991272 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555f1558e9e0 con 0x555f15540300 2024-09-13T16:44:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.584+0000 7ffb9e000640 1 -- 172.21.15.28:0/3444991272 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffbac051900 con 0x555f15540300 2024-09-13T16:44:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.584+0000 7ffbc2218740 10 client.4655 fetch_fsmap learned FSMap version 17 2024-09-13T16:44:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.584+0000 7ffbc2218740 1 -- 172.21.15.28:0/3444991272 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x555f15444470 con 0x555f15540300 2024-09-13T16:44:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.584+0000 7ffb9e000640 1 -- 172.21.15.28:0/3444991272 <== mon.2 v2:172.21.15.191:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555f1558e9e0 con 0x555f15540300 2024-09-13T16:44:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.584+0000 7ffb9e000640 1 -- 172.21.15.28:0/3444991272 <== mon.2 v2:172.21.15.191:3301/0 5 ==== fsmap(e 17) v1 ==== 4252+0+0 (secure 0 0 0) 0x7ffbac051050 con 0x555f15540300 2024-09-13T16:44:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.584+0000 7ffbc2218740 10 client.4655 fetch_fsmap finished waiting for FSMap version 17 2024-09-13T16:44:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.584+0000 7ffbc2218740 10 client.4655 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T16:44:32.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.584+0000 7ffbc2218740 20 client.4655 populate_metadata read hostname 'smithi028' 2024-09-13T16:44:32.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.584+0000 7ffbc2218740 1 --2- 172.21.15.28:0/3444991272 >> [v2:172.21.15.191:6834/704943585,v1:172.21.15.191:6835/704943585] conn(0x555f15576f30 0x555f155e9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:32.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.584+0000 7ffbc2218740 4 client.4655 mds_command: new command op to 4258 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:44:32.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.584+0000 7ffbc2218740 1 -- 172.21.15.28:0/3444991272 --> [v2:172.21.15.191:6834/704943585,v1:172.21.15.191:6835/704943585] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7ffba4054430 con 0x555f15576f30 2024-09-13T16:44:32.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.585+0000 7ffbb7e00640 1 --2- 172.21.15.28:0/3444991272 >> [v2:172.21.15.191:6834/704943585,v1:172.21.15.191:6835/704943585] conn(0x555f15576f30 0x555f155e9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:32.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.585+0000 7ffbb7e00640 1 --2- 172.21.15.28:0/3444991272 >> [v2:172.21.15.191:6834/704943585,v1:172.21.15.191:6835/704943585] conn(0x555f15576f30 0x555f155e9d60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=878ad186b75abaf4 server_cookie=6c043737e17b553b in_seq=0 out_seq=0 2024-09-13T16:44:32.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.585+0000 7ffb9e000640 10 client.4655 ms_handle_connect on v2:172.21.15.191:6834/704943585 2024-09-13T16:44:32.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.585+0000 7ffb9e000640 1 -- 172.21.15.28:0/3444991272 <== mds.0 v2:172.21.15.191:6834/704943585 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x7ffba4054430 con 0x555f15576f30 2024-09-13T16:44:32.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.585+0000 7ffb9e000640 10 client.4655 handle_command_reply: tid=0 2024-09-13T16:44:32.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.585+0000 7ffbc2218740 1 client.4655 shutdown 2024-09-13T16:44:32.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbc2218740 1 -- 172.21.15.28:0/3444991272 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15540300 msgr2=0x555f155f3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbc2218740 1 --2- 172.21.15.28:0/3444991272 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15540300 0x555f155f3a80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0098c0 tx=0x7ffbac02c070 comp rx=0 tx=0).stop 2024-09-13T16:44:32.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbc2218740 1 -- 172.21.15.28:0/3444991272 shutdown_connections 2024-09-13T16:44:32.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbc2218740 1 --2- 172.21.15.28:0/3444991272 >> [v2:172.21.15.191:6834/704943585,v1:172.21.15.191:6835/704943585] conn(0x555f15576f30 0x555f155e9d60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbc2218740 1 --2- 172.21.15.28:0/3444991272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f15542eb0 0x555f155765a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbc2218740 1 --2- 172.21.15.28:0/3444991272 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555f15542ad0 0x555f1557f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbc2218740 1 --2- 172.21.15.28:0/3444991272 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555f15540300 0x555f155f3a80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffb9e000640 0 client.4655 ms_handle_reset on v2:172.21.15.191:6834/704943585 2024-09-13T16:44:32.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbc2218740 1 -- 172.21.15.28:0/3444991272 >> 172.21.15.28:0/3444991272 conn(0x555f155411f0 msgr2=0x555f1556fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbc2218740 1 -- 172.21.15.28:0/3444991272 shutdown_connections 2024-09-13T16:44:32.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbc2218740 1 -- 172.21.15.28:0/3444991272 wait complete. 2024-09-13T16:44:32.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbc2218740 20 client.4655 trim_cache size 0 max 16384 2024-09-13T16:44:32.856 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:44:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:44:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T16:44:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4258, 2024-09-13T16:44:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T16:44:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T16:44:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T16:44:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 37.614478065999997, 2024-09-13T16:44:32.857 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 17, 2024-09-13T16:44:32.860 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 20, 2024-09-13T16:44:32.860 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 20, 2024-09-13T16:44:32.860 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 42.366016684999998 2024-09-13T16:44:32.860 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:44:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbbf400640 1 -- 172.21.15.28:0/246204845 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ffb9806ac30 msgr2=0x7ffb9806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbbf400640 1 --2- 172.21.15.28:0/246204845 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ffb9806ac30 0x7ffb9806d0f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ffbac034000 tx=0x7ffbac02e000 comp rx=0 tx=0).stop 2024-09-13T16:44:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbbf400640 1 -- 172.21.15.28:0/246204845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb80c4d60 msgr2=0x7ffbb8147fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbbf400640 1 --2- 172.21.15.28:0/246204845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb80c4d60 0x7ffbb8147fe0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffba400c490 tx=0x7ffba400c950 comp rx=0 tx=0).stop 2024-09-13T16:44:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbbf400640 1 -- 172.21.15.28:0/246204845 shutdown_connections 2024-09-13T16:44:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbbf400640 1 --2- 172.21.15.28:0/246204845 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ffb9806ac30 0x7ffb9806d0f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbbf400640 1 --2- 172.21.15.28:0/246204845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbb8150ca0 0x7ffbb80bfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbbf400640 1 --2- 172.21.15.28:0/246204845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbb80c4d60 0x7ffbb8147fe0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbbf400640 1 --2- 172.21.15.28:0/246204845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbb80c2250 0x7ffbb814c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.586+0000 7ffbbf400640 1 -- 172.21.15.28:0/246204845 >> 172.21.15.28:0/246204845 conn(0x7ffbb8066040 msgr2=0x7ffbb80c5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.587+0000 7ffbbf400640 1 -- 172.21.15.28:0/246204845 shutdown_connections 2024-09-13T16:44:32.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:32.587+0000 7ffbbf400640 1 -- 172.21.15.28:0/246204845 wait complete. 2024-09-13T16:44:32.864 INFO:teuthology.run_tasks:Running task sequential... 2024-09-13T16:44:32.881 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-09-13T16:44:32.881 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-13T16:44:32.882 INFO:teuthology.task.print:Enabling mgr modules 2024-09-13T16:44:32.882 INFO:teuthology.run_tasks:Running task kclient... 2024-09-13T16:44:32.908 INFO:tasks.kclient:Mounting kernel clients... 2024-09-13T16:44:32.908 INFO:tasks.kclient:config is {'client.0': {}} 2024-09-13T16:44:32.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:32.909 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T16:44:32.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:32.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-09-13T16:44:33.046 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-09-13T16:44:33.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:44:33.050 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-09-13T16:44:33.050 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-13T16:44:33.050 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:44:33.051 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:44:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.203+0000 7f292da00640 1 Processor -- start 2024-09-13T16:44:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.204+0000 7f292da00640 1 -- start start 2024-09-13T16:44:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.205+0000 7f292da00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f29280c1b40 0x7f2928156990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.205+0000 7f292da00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29280c6240 0x7f2928156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.205+0000 7f292da00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f292806e100 con 0x7f29281509b0 2024-09-13T16:44:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.205+0000 7f292da00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2928150d90 con 0x7f29280c1b40 2024-09-13T16:44:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.205+0000 7f292da00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29280c43b0 con 0x7f29280c6240 2024-09-13T16:44:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.205+0000 7f2927e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f29281509b0 0x7f292806cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53778/0 (socket says 172.21.15.28:53778) 2024-09-13T16:44:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.205+0000 7f2927e00640 1 -- 172.21.15.28:0/3922206331 learned_addr learned my addr 172.21.15.28:0/3922206331 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.205+0000 7f291f400640 1 --2- 172.21.15.28:0/3922206331 >> v2:172.21.15.191:3300/0 conn(0x7f29280c1b40 0x7f2928156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.205+0000 7f292d000640 1 --2- 172.21.15.28:0/3922206331 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29280c6240 0x7f2928156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.206+0000 7f2927400640 1 -- 172.21.15.28:0/3922206331 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815712558 0 0) 0x7f292806e100 con 0x7f29281509b0 2024-09-13T16:44:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.206+0000 7f2927400640 1 -- 172.21.15.28:0/3922206331 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f290c003520 con 0x7f29281509b0 2024-09-13T16:44:33.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.206+0000 7f292d000640 1 -- 172.21.15.28:0/3922206331 >> v2:172.21.15.191:3300/0 conn(0x7f29280c1b40 msgr2=0x7f2928156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.206+0000 7f292d000640 1 --2- 172.21.15.28:0/3922206331 >> v2:172.21.15.191:3300/0 conn(0x7f29280c1b40 0x7f2928156990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.206+0000 7f292d000640 1 -- 172.21.15.28:0/3922206331 >> v1:172.21.15.28:6789/0 conn(0x7f29281509b0 legacy=0x7f292806cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.206+0000 7f292d000640 1 -- 172.21.15.28:0/3922206331 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292806d7d0 con 0x7f29280c6240 2024-09-13T16:44:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.207+0000 7f292d000640 1 --2- 172.21.15.28:0/3922206331 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29280c6240 0x7f2928156ed0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2920009d90 tx=0x7f292002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c392bfa0a07ddefe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.207+0000 7f2927400640 1 -- 172.21.15.28:0/3922206331 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2920007740 con 0x7f29280c6240 2024-09-13T16:44:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.207+0000 7f2927400640 1 -- 172.21.15.28:0/3922206331 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f292002d050 con 0x7f29280c6240 2024-09-13T16:44:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.207+0000 7f2927400640 1 -- 172.21.15.28:0/3922206331 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2920007ac0 con 0x7f29280c6240 2024-09-13T16:44:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.207+0000 7f292da00640 1 -- 172.21.15.28:0/3922206331 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29280c6240 msgr2=0x7f2928156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.207+0000 7f292da00640 1 --2- 172.21.15.28:0/3922206331 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29280c6240 0x7f2928156ed0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2920009d90 tx=0x7f292002a660 comp rx=0 tx=0).stop 2024-09-13T16:44:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.208+0000 7f292da00640 1 -- 172.21.15.28:0/3922206331 shutdown_connections 2024-09-13T16:44:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.208+0000 7f292da00640 1 --2- 172.21.15.28:0/3922206331 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29280c6240 0x7f2928156ed0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.208+0000 7f292da00640 1 --2- 172.21.15.28:0/3922206331 >> v2:172.21.15.191:3300/0 conn(0x7f29280c1b40 0x7f2928156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.208+0000 7f292da00640 1 -- 172.21.15.28:0/3922206331 >> 172.21.15.28:0/3922206331 conn(0x7f29280ca0e0 msgr2=0x7f29280ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.208+0000 7f292da00640 1 -- 172.21.15.28:0/3922206331 shutdown_connections 2024-09-13T16:44:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.208+0000 7f292da00640 1 -- 172.21.15.28:0/3922206331 wait complete. 2024-09-13T16:44:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.209+0000 7f292da00640 1 Processor -- start 2024-09-13T16:44:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.209+0000 7f292da00640 1 -- start start 2024-09-13T16:44:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.210+0000 7f292da00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29280c1b40 0x7f29281f2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.210+0000 7f292da00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29280c6240 0x7f2928070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.210+0000 7f292da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29281509b0 0x7f2928148cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.210+0000 7f292da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29281558d0 con 0x7f29281509b0 2024-09-13T16:44:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.210+0000 7f292da00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29281f2760 con 0x7f29280c1b40 2024-09-13T16:44:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.210+0000 7f292da00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29281484e0 con 0x7f29280c6240 2024-09-13T16:44:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.211+0000 7f292d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29281509b0 0x7f2928148cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.211+0000 7f2927e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29280c1b40 0x7f29281f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.211+0000 7f291f400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29280c6240 0x7f2928070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.211+0000 7f291f400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29280c6240 0x7f2928070f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:55382/0 (socket says 172.21.15.28:55382) 2024-09-13T16:44:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.211+0000 7f291f400640 1 -- 172.21.15.28:0/775175008 learned_addr learned my addr 172.21.15.28:0/775175008 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.211+0000 7f291f400640 1 -- 172.21.15.28:0/775175008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29280c1b40 msgr2=0x7f29281f2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.211+0000 7f291f400640 1 --2- 172.21.15.28:0/775175008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29280c1b40 0x7f29281f2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.211+0000 7f291f400640 1 -- 172.21.15.28:0/775175008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29281509b0 msgr2=0x7f2928148cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.211+0000 7f291f400640 1 --2- 172.21.15.28:0/775175008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29281509b0 0x7f2928148cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.211+0000 7f291f400640 1 -- 172.21.15.28:0/775175008 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2920009a70 con 0x7f29280c6240 2024-09-13T16:44:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.211+0000 7f2927e00640 1 --2- 172.21.15.28:0/775175008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29280c1b40 0x7f29281f2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.211+0000 7f292d000640 1 --2- 172.21.15.28:0/775175008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29281509b0 0x7f2928148cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.211+0000 7f291f400640 1 --2- 172.21.15.28:0/775175008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29280c6240 0x7f2928070f90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f291000e9b0 tx=0x7f291000ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.211+0000 7f2924c00640 1 -- 172.21.15.28:0/775175008 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2910009670 con 0x7f29280c6240 2024-09-13T16:44:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.212+0000 7f2924c00640 1 -- 172.21.15.28:0/775175008 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291000c9a0 con 0x7f29280c6240 2024-09-13T16:44:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.212+0000 7f2924c00640 1 -- 172.21.15.28:0/775175008 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291000cd50 con 0x7f29280c6240 2024-09-13T16:44:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.212+0000 7f292da00640 1 -- 172.21.15.28:0/775175008 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2928155b00 con 0x7f29280c6240 2024-09-13T16:44:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.212+0000 7f292da00640 1 -- 172.21.15.28:0/775175008 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29281560c0 con 0x7f29280c6240 2024-09-13T16:44:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.213+0000 7f292da00640 1 -- 172.21.15.28:0/775175008 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29280c2a00 con 0x7f29280c6240 2024-09-13T16:44:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.214+0000 7f2924c00640 1 -- 172.21.15.28:0/775175008 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2910013470 con 0x7f29280c6240 2024-09-13T16:44:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.215+0000 7f2924c00640 1 --2- 172.21.15.28:0/775175008 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f290406ac00 0x7f290406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.215+0000 7f2924c00640 1 -- 172.21.15.28:0/775175008 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2910089670 con 0x7f29280c6240 2024-09-13T16:44:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.215+0000 7f2927e00640 1 --2- 172.21.15.28:0/775175008 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f290406ac00 0x7f290406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:33.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.215+0000 7f2927e00640 1 --2- 172.21.15.28:0/775175008 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f290406ac00 0x7f290406d0c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f29180011e0 tx=0x7f2918007d30 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:33.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.219+0000 7f2924c00640 1 -- 172.21.15.28:0/775175008 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f29100547d0 con 0x7f29280c6240 2024-09-13T16:44:33.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:44:33.359 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:44:33.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.434+0000 7f292da00640 1 -- 172.21.15.28:0/775175008 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f29280c3d80 con 0x7f29280c6240 2024-09-13T16:44:33.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.435+0000 7f2924c00640 1 -- 172.21.15.28:0/775175008 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2910057df0 con 0x7f29280c6240 2024-09-13T16:44:33.437 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T16:44:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.437+0000 7f292da00640 1 -- 172.21.15.28:0/775175008 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f290406ac00 msgr2=0x7f290406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.437+0000 7f292da00640 1 --2- 172.21.15.28:0/775175008 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f290406ac00 0x7f290406d0c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f29180011e0 tx=0x7f2918007d30 comp rx=0 tx=0).stop 2024-09-13T16:44:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.437+0000 7f292da00640 1 -- 172.21.15.28:0/775175008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29280c6240 msgr2=0x7f2928070f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.437+0000 7f292da00640 1 --2- 172.21.15.28:0/775175008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29280c6240 0x7f2928070f90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f291000e9b0 tx=0x7f291000ee70 comp rx=0 tx=0).stop 2024-09-13T16:44:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.437+0000 7f292da00640 1 -- 172.21.15.28:0/775175008 shutdown_connections 2024-09-13T16:44:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.437+0000 7f292da00640 1 --2- 172.21.15.28:0/775175008 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f290406ac00 0x7f290406d0c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.437+0000 7f292da00640 1 --2- 172.21.15.28:0/775175008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29281509b0 0x7f2928148cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.437+0000 7f292da00640 1 --2- 172.21.15.28:0/775175008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29280c6240 0x7f2928070f90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.437+0000 7f292da00640 1 --2- 172.21.15.28:0/775175008 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29280c1b40 0x7f29281f2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.437+0000 7f292da00640 1 -- 172.21.15.28:0/775175008 >> 172.21.15.28:0/775175008 conn(0x7f29280ca0e0 msgr2=0x7f292806c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.438+0000 7f292da00640 1 -- 172.21.15.28:0/775175008 shutdown_connections 2024-09-13T16:44:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:33.438+0000 7f292da00640 1 -- 172.21.15.28:0/775175008 wait complete. 2024-09-13T16:44:33.455 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:44:33.455 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:44:33.455 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T16:44:33.455 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T16:44:33.455 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:44:33.455 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:44:33.455 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:44:33.455 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:44:33.455 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:44:33.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:33.455 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T16:44:33.531 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:44:33.531 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:44:33.531 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:44:33.532 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:44:33.532 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T16:44:33.532 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:44:33.532 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T16:44:33.532 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:44:33.532 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T16:44:33.532 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T16:44:33.532 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:44:33.532 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T16:44:33.533 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:44:33.533 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T16:44:33.533 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T16:44:33.533 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T16:44:33.533 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42706sec preferred_lft 42706sec 2024-09-13T16:44:33.533 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T16:44:33.533 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:44:33.533 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T16:44:33.533 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T16:44:33.534 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T16:44:33.534 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-13T16:44:33.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:33.535 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T16:44:33.535 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T16:44:33.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-09-13T16:44:33.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-09-13T16:44:33.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-09-13T16:44:33.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-13T16:44:33.536 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T16:44:33.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:33.917 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:44:33.975 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T16:44:33.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:33.979 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T16:44:34.025 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T16:44:34.025 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:44:34.025 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T16:44:34.025 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T16:44:34.025 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:44:34.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:34.026 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T16:44:34.026 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T16:44:34.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T16:44:34.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T16:44:34.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T16:44:34.026 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T16:44:34.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:34.597 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T16:44:34.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:34.634 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-09-13T16:44:34.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:34.711 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T16:44:34.711 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T16:44:34.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:44:34.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-13T16:44:34.712 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T16:44:34.849 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-13T16:44:34.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:34.849 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T16:44:34.849 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T16:44:34.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-13T16:44:34.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-13T16:44:34.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-13T16:44:34.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-13T16:44:34.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-13T16:44:34.850 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T16:44:35.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:35.054 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T16:44:35.054 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T16:44:35.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T16:44:35.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-09-13T16:44:35.055 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T16:44:35.171 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:44:35.172 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T16:44:35.207 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:44:35.207 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:44:35.207 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:44:35.281 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T16:44:35.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:35.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T16:44:35.417 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T16:44:35.417 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T16:44:36.031 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T16:44:36.032 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T16:44:36.032 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.191:3300,172.21.15.191:3301" 2024-09-13T16:44:36.032 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@8e702092-2b7f-4633-9ce6-17040007e53d.cephfs=/ 2024-09-13T16:44:36.032 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.191:3300/172.21.15.191:3301" will pass to kernel 2024-09-13T16:44:36.035 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T16:44:36.035 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T16:44:36.035 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T16:44:36.035 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T16:44:36.035 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T16:44:36.038 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T16:44:36.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:36.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:44:36.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:36.100 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T16:44:36.146 INFO:teuthology.orchestra.run.smithi028.stdout:client4713 2024-09-13T16:44:36.147 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:44:36.147 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:44:36.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:36.223 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T16:44:36.307 INFO:teuthology.orchestra.run.smithi028.stdout:client4713 2024-09-13T16:44:36.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:36.308 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T16:44:36.393 INFO:teuthology.orchestra.run.smithi028.stdout:client4713 2024-09-13T16:44:36.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:44:36.394 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/8e702092-2b7f-4633-9ce6-17040007e53d.client4713/status') 2024-09-13T16:44:36.493 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.4713 (3)192.168.0.1:0/4069375352 2024-09-13T16:44:36.494 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-09-13T16:44:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-09-13T16:44:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-09-13T16:44:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:69 bytes copied, 5.8686e-05 s, 1.2 MB/s 2024-09-13T16:44:36.496 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-13T16:44:36.508 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-13T16:44:36.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:44:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.662+0000 7f7724000640 1 Processor -- start 2024-09-13T16:44:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.662+0000 7f7724000640 1 -- start start 2024-09-13T16:44:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.663+0000 7f7724000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f771c150ca0 0x7f771c1568e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.663+0000 7f7724000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c4d60 0x7f771c0c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.663+0000 7f7724000640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771c151080 con 0x7f771c0c2250 2024-09-13T16:44:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.663+0000 7f7724000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f771c0c2670 con 0x7f771c150ca0 2024-09-13T16:44:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.663+0000 7f7724000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771c15ae40 con 0x7f771c0c4d60 2024-09-13T16:44:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.663+0000 7f7721600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f771c150ca0 0x7f771c1568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.664+0000 7f7721600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f771c150ca0 0x7f771c1568e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44268/0 (socket says 172.21.15.28:44268) 2024-09-13T16:44:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.664+0000 7f7721600640 1 -- 172.21.15.28:0/71662192 learned_addr learned my addr 172.21.15.28:0/71662192 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.664+0000 7f7722000640 1 --2- 172.21.15.28:0/71662192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c4d60 0x7f771c0c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.664+0000 7f7721600640 1 -- 172.21.15.28:0/71662192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c4d60 msgr2=0x7f771c0c51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.664+0000 7f7721600640 1 --2- 172.21.15.28:0/71662192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c4d60 0x7f771c0c51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.664+0000 7f7721600640 1 -- 172.21.15.28:0/71662192 >> v1:172.21.15.191:6789/0 conn(0x7f771c0c2250 legacy=0x7f771c15a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.664+0000 7f7721600640 1 -- 172.21.15.28:0/71662192 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771c06a990 con 0x7f771c150ca0 2024-09-13T16:44:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.664+0000 7f7722000640 1 --2- 172.21.15.28:0/71662192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c4d60 0x7f771c0c51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.665+0000 7f7721600640 1 --2- 172.21.15.28:0/71662192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f771c150ca0 0x7f771c1568e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f77100098a0 tx=0x7f771002a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e043cc600531ebed server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.665+0000 7f771be00640 1 -- 172.21.15.28:0/71662192 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710032440 con 0x7f771c150ca0 2024-09-13T16:44:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.665+0000 7f771be00640 1 -- 172.21.15.28:0/71662192 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77100325a0 con 0x7f771c150ca0 2024-09-13T16:44:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.665+0000 7f771be00640 1 -- 172.21.15.28:0/71662192 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710032930 con 0x7f771c150ca0 2024-09-13T16:44:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.666+0000 7f7724000640 1 -- 172.21.15.28:0/71662192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f771c150ca0 msgr2=0x7f771c1568e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.666+0000 7f7724000640 1 --2- 172.21.15.28:0/71662192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f771c150ca0 0x7f771c1568e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f77100098a0 tx=0x7f771002a200 comp rx=0 tx=0).stop 2024-09-13T16:44:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.666+0000 7f7724000640 1 -- 172.21.15.28:0/71662192 shutdown_connections 2024-09-13T16:44:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.666+0000 7f7724000640 1 --2- 172.21.15.28:0/71662192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c4d60 0x7f771c0c51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.666+0000 7f7724000640 1 --2- 172.21.15.28:0/71662192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f771c150ca0 0x7f771c1568e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.666+0000 7f7724000640 1 -- 172.21.15.28:0/71662192 >> 172.21.15.28:0/71662192 conn(0x7f771c066040 msgr2=0x7f771c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.666+0000 7f7724000640 1 -- 172.21.15.28:0/71662192 shutdown_connections 2024-09-13T16:44:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.666+0000 7f7724000640 1 -- 172.21.15.28:0/71662192 wait complete. 2024-09-13T16:44:36.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.668+0000 7f7724000640 1 Processor -- start 2024-09-13T16:44:36.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.668+0000 7f7724000640 1 -- start start 2024-09-13T16:44:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.669+0000 7f7724000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c2250 0x7f771c1ec390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.669+0000 7f7724000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f771c0c4d60 0x7f771c1f0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.669+0000 7f7724000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f771c150ca0 0x7f771c13c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.669+0000 7f7724000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c1ef900 con 0x7f771c150ca0 2024-09-13T16:44:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.669+0000 7f7724000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c13c5b0 con 0x7f771c0c4d60 2024-09-13T16:44:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.669+0000 7f7724000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771c1ed320 con 0x7f771c0c2250 2024-09-13T16:44:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.669+0000 7f7722000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f771c150ca0 0x7f771c13c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.669+0000 7f7722000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f771c150ca0 0x7f771c13c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44278/0 (socket says 172.21.15.28:44278) 2024-09-13T16:44:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.669+0000 7f7722000640 1 -- 172.21.15.28:0/352513075 learned_addr learned my addr 172.21.15.28:0/352513075 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.669+0000 7f7720c00640 1 --2- 172.21.15.28:0/352513075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f771c0c4d60 0x7f771c1f0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.669+0000 7f7721600640 1 --2- 172.21.15.28:0/352513075 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c2250 0x7f771c1ec390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.670+0000 7f7722000640 1 -- 172.21.15.28:0/352513075 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c2250 msgr2=0x7f771c1ec390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.670+0000 7f7722000640 1 --2- 172.21.15.28:0/352513075 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c2250 0x7f771c1ec390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.670+0000 7f7722000640 1 -- 172.21.15.28:0/352513075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f771c0c4d60 msgr2=0x7f771c1f0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.670+0000 7f7722000640 1 --2- 172.21.15.28:0/352513075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f771c0c4d60 0x7f771c1f0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.670+0000 7f7722000640 1 -- 172.21.15.28:0/352513075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77100094d0 con 0x7f771c150ca0 2024-09-13T16:44:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.670+0000 7f7720c00640 1 --2- 172.21.15.28:0/352513075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f771c0c4d60 0x7f771c1f0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.670+0000 7f7722000640 1 --2- 172.21.15.28:0/352513075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f771c150ca0 0x7f771c13c070 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7714007f70 tx=0x7f771400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.670+0000 7f7719600640 1 -- 172.21.15.28:0/352513075 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714003b90 con 0x7f771c150ca0 2024-09-13T16:44:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.670+0000 7f7719600640 1 -- 172.21.15.28:0/352513075 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771400f040 con 0x7f771c150ca0 2024-09-13T16:44:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.670+0000 7f7719600640 1 -- 172.21.15.28:0/352513075 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714004030 con 0x7f771c150ca0 2024-09-13T16:44:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.670+0000 7f7724000640 1 -- 172.21.15.28:0/352513075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771c157e10 con 0x7f771c150ca0 2024-09-13T16:44:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.671+0000 7f7724000640 1 -- 172.21.15.28:0/352513075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c1583d0 con 0x7f771c150ca0 2024-09-13T16:44:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.672+0000 7f7724000640 1 -- 172.21.15.28:0/352513075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f771c0c5500 con 0x7f771c150ca0 2024-09-13T16:44:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.672+0000 7f7719600640 1 -- 172.21.15.28:0/352513075 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7714004190 con 0x7f771c150ca0 2024-09-13T16:44:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.673+0000 7f7719600640 1 --2- 172.21.15.28:0/352513075 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f76fc06ac00 0x7f76fc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.673+0000 7f7719600640 1 -- 172.21.15.28:0/352513075 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7714089720 con 0x7f771c150ca0 2024-09-13T16:44:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.674+0000 7f7721600640 1 --2- 172.21.15.28:0/352513075 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f76fc06ac00 0x7f76fc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.674+0000 7f7721600640 1 --2- 172.21.15.28:0/352513075 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f76fc06ac00 0x7f76fc06d0c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7710034000 tx=0x7f771002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:36.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.678+0000 7f7719600640 1 -- 172.21.15.28:0/352513075 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7714054910 con 0x7f771c150ca0 2024-09-13T16:44:36.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.858+0000 7f7724000640 1 -- 172.21.15.28:0/352513075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f771c15a210 con 0x7f771c150ca0 2024-09-13T16:44:36.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.860+0000 7f7719600640 1 -- 172.21.15.28:0/352513075 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16481 (secure 0 0 0) 0x7f7714057f30 con 0x7f771c150ca0 2024-09-13T16:44:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:36.861 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:43:55.971358+0000","last_up_change":"2024-09-13T16:43:49.476313+0000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:43:49.643606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:43:52.828510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:43:53.945874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6801","nonce":2964323435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6803","nonce":2964323435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6807","nonce":2964323435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6805","nonce":2964323435}]},"public_addr":"172.21.15.28:6801/2964323435","cluster_addr":"172.21.15.28:6803/2964323435","heartbeat_back_addr":"172.21.15.28:6807/2964323435","heartbeat_front_addr":"172.21.15.28:6805/2964323435","state":["exists","up"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6809","nonce":23768570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6811","nonce":23768570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6815","nonce":23768570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6813","nonce":23768570}]},"public_addr":"172.21.15.28:6809/23768570","cluster_addr":"172.21.15.28:6811/23768570","heartbeat_back_addr":"172.21.15.28:6815/23768570","heartbeat_front_addr":"172.21.15.28:6813/23768570","state":["exists","up"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6817","nonce":886887303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6819","nonce":886887303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6823","nonce":886887303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6821","nonce":886887303}]},"public_addr":"172.21.15.28:6817/886887303","cluster_addr":"172.21.15.28:6819/886887303","heartbeat_back_addr":"172.21.15.28:6823/886887303","heartbeat_front_addr":"172.21.15.28:6821/886887303","state":["exists","up"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6825","nonce":2616036674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6827","nonce":2616036674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6831","nonce":2616036674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6829","nonce":2616036674}]},"public_addr":"172.21.15.28:6825/2616036674","cluster_addr":"172.21.15.28:6827/2616036674","heartbeat_back_addr":"172.21.15.28:6831/2616036674","heartbeat_front_addr":"172.21.15.28:6829/2616036674","state":["exists","up"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6801","nonce":2817102450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6803","nonce":2817102450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6807","nonce":2817102450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6805","nonce":2817102450}]},"public_addr":"172.21.15.191:6801/2817102450","cluster_addr":"172.21.15.191:6803/2817102450","heartbeat_back_addr":"172.21.15.191:6807/2817102450","heartbeat_front_addr":"172.21.15.191:6805/2817102450","state":["exists","up"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6817","nonce":2503027971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6819","nonce":2503027971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6823","nonce":2503027971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6821","nonce":2503027971}]},"public_addr":"172.21.15.191:6817/2503027971","cluster_addr":"172.21.15.191:6819/2503027971","heartbeat_back_addr":"172.21.15.191:6823/2503027971","heartbeat_front_addr":"172.21.15.191:6821/2503027971","state":["exists","up"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6809","nonce":3006595088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6811","nonce":3006595088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6815","nonce":3006595088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6813","nonce":3006595088}]},"public_addr":"172.21.15.191:6809/3006595088","cluster_addr":"172.21.15.191:6811/3006595088","heartbeat_back_addr":"172.21.15.191:6815/3006595088","heartbeat_front_addr":"172.21.15.191:6813/3006595088","state":["exists","up"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6825","nonce":1477327720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6827","nonce":1477327720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6831","nonce":1477327720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6829","nonce":1477327720}]},"public_addr":"172.21.15.191:6825/1477327720","cluster_addr":"172.21.15.191:6827/1477327720","heartbeat_back_addr":"172.21.15.191:6831/1477327720","heartbeat_front_addr":"172.21.15.191:6829/1477327720","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.917250+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.954351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.017329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.046221+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.707467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.726947+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.782321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.764850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:44:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.862+0000 7f7724000640 1 -- 172.21.15.28:0/352513075 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f76fc06ac00 msgr2=0x7f76fc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.862+0000 7f7724000640 1 --2- 172.21.15.28:0/352513075 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f76fc06ac00 0x7f76fc06d0c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7710034000 tx=0x7f771002e000 comp rx=0 tx=0).stop 2024-09-13T16:44:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.862+0000 7f7724000640 1 -- 172.21.15.28:0/352513075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f771c150ca0 msgr2=0x7f771c13c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.862+0000 7f7724000640 1 --2- 172.21.15.28:0/352513075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f771c150ca0 0x7f771c13c070 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7714007f70 tx=0x7f771400d560 comp rx=0 tx=0).stop 2024-09-13T16:44:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.862+0000 7f7724000640 1 -- 172.21.15.28:0/352513075 shutdown_connections 2024-09-13T16:44:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.862+0000 7f7724000640 1 --2- 172.21.15.28:0/352513075 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f76fc06ac00 0x7f76fc06d0c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.862+0000 7f7724000640 1 --2- 172.21.15.28:0/352513075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f771c150ca0 0x7f771c13c070 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.862+0000 7f7724000640 1 --2- 172.21.15.28:0/352513075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f771c0c4d60 0x7f771c1f0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.862+0000 7f7724000640 1 --2- 172.21.15.28:0/352513075 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c2250 0x7f771c1ec390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.862+0000 7f7724000640 1 -- 172.21.15.28:0/352513075 >> 172.21.15.28:0/352513075 conn(0x7f771c066040 msgr2=0x7f771c146980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.862+0000 7f7724000640 1 -- 172.21.15.28:0/352513075 shutdown_connections 2024-09-13T16:44:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:36.862+0000 7f7724000640 1 -- 172.21.15.28:0/352513075 wait complete. 2024-09-13T16:44:36.879 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:43:49.643606+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:43:52.828510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:43:53.945874+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:44:36.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:44:37.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.036+0000 7f6f4d800640 1 Processor -- start 2024-09-13T16:44:37.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.036+0000 7f6f4d800640 1 -- start start 2024-09-13T16:44:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.037+0000 7f6f4d800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f48150b30 0x7f6f480c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.037+0000 7f6f4d800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6f480c9970 0x7f6f481610f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.037+0000 7f6f4d800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4806e7f0 con 0x7f6f480cafb0 2024-09-13T16:44:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.037+0000 7f6f4d800640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f480c5ac0 con 0x7f6f480c9970 2024-09-13T16:44:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.037+0000 7f6f4d800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f480c9e20 con 0x7f6f48150b30 2024-09-13T16:44:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.038+0000 7f6f46000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6f480cafb0 0x7f6f480c3a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53786/0 (socket says 172.21.15.28:53786) 2024-09-13T16:44:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.038+0000 7f6f46000640 1 -- 172.21.15.28:0/1426984279 learned_addr learned my addr 172.21.15.28:0/1426984279 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.038+0000 7f6f46a00640 1 --2- 172.21.15.28:0/1426984279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f48150b30 0x7f6f480c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.038+0000 7f6f47400640 1 --2- 172.21.15.28:0/1426984279 >> v2:172.21.15.191:3300/0 conn(0x7f6f480c9970 0x7f6f481610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.038+0000 7f6f45600640 1 -- 172.21.15.28:0/1426984279 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023157400 0 0) 0x7f6f4806e7f0 con 0x7f6f480cafb0 2024-09-13T16:44:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.038+0000 7f6f45600640 1 -- 172.21.15.28:0/1426984279 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f2c003520 con 0x7f6f480cafb0 2024-09-13T16:44:37.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.038+0000 7f6f47400640 1 -- 172.21.15.28:0/1426984279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f48150b30 msgr2=0x7f6f480c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:37.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.038+0000 7f6f47400640 1 --2- 172.21.15.28:0/1426984279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f48150b30 0x7f6f480c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.038+0000 7f6f47400640 1 -- 172.21.15.28:0/1426984279 >> v1:172.21.15.28:6789/0 conn(0x7f6f480cafb0 legacy=0x7f6f480c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.038+0000 7f6f47400640 1 -- 172.21.15.28:0/1426984279 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4806e570 con 0x7f6f480c9970 2024-09-13T16:44:37.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.038+0000 7f6f46a00640 1 --2- 172.21.15.28:0/1426984279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f48150b30 0x7f6f480c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:37.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.039+0000 7f6f47400640 1 --2- 172.21.15.28:0/1426984279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f480c9970 0x7f6f481610f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6f40009d90 tx=0x7f6f4002a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9607b14ccdf60cb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:37.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.040+0000 7f6f45600640 1 -- 172.21.15.28:0/1426984279 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40007740 con 0x7f6f480c9970 2024-09-13T16:44:37.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.040+0000 7f6f45600640 1 -- 172.21.15.28:0/1426984279 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4002d050 con 0x7f6f480c9970 2024-09-13T16:44:37.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.040+0000 7f6f45600640 1 -- 172.21.15.28:0/1426984279 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40007ac0 con 0x7f6f480c9970 2024-09-13T16:44:37.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.040+0000 7f6f4d800640 1 -- 172.21.15.28:0/1426984279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f480c9970 msgr2=0x7f6f481610f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.040+0000 7f6f4d800640 1 --2- 172.21.15.28:0/1426984279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f480c9970 0x7f6f481610f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6f40009d90 tx=0x7f6f4002a6f0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.041+0000 7f6f4d800640 1 -- 172.21.15.28:0/1426984279 shutdown_connections 2024-09-13T16:44:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.041+0000 7f6f4d800640 1 --2- 172.21.15.28:0/1426984279 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f480c9970 0x7f6f481610f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.041+0000 7f6f4d800640 1 --2- 172.21.15.28:0/1426984279 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f48150b30 0x7f6f480c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.041+0000 7f6f4d800640 1 -- 172.21.15.28:0/1426984279 >> 172.21.15.28:0/1426984279 conn(0x7f6f480ca2f0 msgr2=0x7f6f480ca6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.041+0000 7f6f4d800640 1 -- 172.21.15.28:0/1426984279 shutdown_connections 2024-09-13T16:44:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.041+0000 7f6f4d800640 1 -- 172.21.15.28:0/1426984279 wait complete. 2024-09-13T16:44:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.042+0000 7f6f4d800640 1 Processor -- start 2024-09-13T16:44:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.042+0000 7f6f4d800640 1 -- start start 2024-09-13T16:44:37.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.043+0000 7f6f4d800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f480c9970 0x7f6f48149640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:37.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.043+0000 7f6f4d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f480cafb0 0x7f6f48151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:37.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.043+0000 7f6f4d800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f48150b30 0x7f6f48153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.043+0000 7f6f4d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f481dda20 con 0x7f6f480cafb0 2024-09-13T16:44:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.043+0000 7f6f4d800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f481e43d0 con 0x7f6f48150b30 2024-09-13T16:44:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.043+0000 7f6f4d800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f481e46b0 con 0x7f6f480c9970 2024-09-13T16:44:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.044+0000 7f6f46000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f480cafb0 0x7f6f48151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.044+0000 7f6f46a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f480c9970 0x7f6f48149640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.044+0000 7f6f46a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f480c9970 0x7f6f48149640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:55412/0 (socket says 172.21.15.28:55412) 2024-09-13T16:44:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.044+0000 7f6f46a00640 1 -- 172.21.15.28:0/2984640521 learned_addr learned my addr 172.21.15.28:0/2984640521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.044+0000 7f6f47400640 1 --2- 172.21.15.28:0/2984640521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f48150b30 0x7f6f48153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.044+0000 7f6f46a00640 1 -- 172.21.15.28:0/2984640521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f48150b30 msgr2=0x7f6f48153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.044+0000 7f6f46a00640 1 --2- 172.21.15.28:0/2984640521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f48150b30 0x7f6f48153c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.044+0000 7f6f46a00640 1 -- 172.21.15.28:0/2984640521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f480cafb0 msgr2=0x7f6f48151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.044+0000 7f6f46a00640 1 --2- 172.21.15.28:0/2984640521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f480cafb0 0x7f6f48151180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.044+0000 7f6f46a00640 1 -- 172.21.15.28:0/2984640521 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f38009400 con 0x7f6f480c9970 2024-09-13T16:44:37.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.044+0000 7f6f47400640 1 --2- 172.21.15.28:0/2984640521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f48150b30 0x7f6f48153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:37.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.045+0000 7f6f46a00640 1 --2- 172.21.15.28:0/2984640521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f480c9970 0x7f6f48149640 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6f3800e960 tx=0x7f6f3800ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:37.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.045+0000 7f6f3ea00640 1 -- 172.21.15.28:0/2984640521 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3800ca00 con 0x7f6f480c9970 2024-09-13T16:44:37.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.045+0000 7f6f3ea00640 1 -- 172.21.15.28:0/2984640521 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3800cb60 con 0x7f6f480c9970 2024-09-13T16:44:37.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.045+0000 7f6f3ea00640 1 -- 172.21.15.28:0/2984640521 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f38013470 con 0x7f6f480c9970 2024-09-13T16:44:37.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.045+0000 7f6f4d800640 1 -- 172.21.15.28:0/2984640521 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f40009a70 con 0x7f6f480c9970 2024-09-13T16:44:37.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.045+0000 7f6f4d800640 1 -- 172.21.15.28:0/2984640521 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f481eecf0 con 0x7f6f480c9970 2024-09-13T16:44:37.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.046+0000 7f6f4d800640 1 -- 172.21.15.28:0/2984640521 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f480c5350 con 0x7f6f480c9970 2024-09-13T16:44:37.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.047+0000 7f6f3ea00640 1 -- 172.21.15.28:0/2984640521 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6f380135d0 con 0x7f6f480c9970 2024-09-13T16:44:37.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.048+0000 7f6f3ea00640 1 --2- 172.21.15.28:0/2984640521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6f2806ac30 0x7f6f2806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:37.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.048+0000 7f6f3ea00640 1 -- 172.21.15.28:0/2984640521 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f38089920 con 0x7f6f480c9970 2024-09-13T16:44:37.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.048+0000 7f6f46000640 1 --2- 172.21.15.28:0/2984640521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6f2806ac30 0x7f6f2806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:37.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.049+0000 7f6f46000640 1 --2- 172.21.15.28:0/2984640521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6f2806ac30 0x7f6f2806d0f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6f481565d0 tx=0x7f6f34007d30 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:37.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.052+0000 7f6f3ea00640 1 -- 172.21.15.28:0/2984640521 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6f38054b10 con 0x7f6f480c9970 2024-09-13T16:44:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.227+0000 7f6f4d800640 1 -- 172.21.15.28:0/2984640521 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6f4814f3f0 con 0x7f6f480c9970 2024-09-13T16:44:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.228+0000 7f6f3ea00640 1 -- 172.21.15.28:0/2984640521 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6f38058130 con 0x7f6f480c9970 2024-09-13T16:44:37.230 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T16:44:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.230+0000 7f6f4d800640 1 -- 172.21.15.28:0/2984640521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6f2806ac30 msgr2=0x7f6f2806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.230+0000 7f6f4d800640 1 --2- 172.21.15.28:0/2984640521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6f2806ac30 0x7f6f2806d0f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6f481565d0 tx=0x7f6f34007d30 comp rx=0 tx=0).stop 2024-09-13T16:44:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.230+0000 7f6f4d800640 1 -- 172.21.15.28:0/2984640521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f480c9970 msgr2=0x7f6f48149640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.230+0000 7f6f4d800640 1 --2- 172.21.15.28:0/2984640521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f480c9970 0x7f6f48149640 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6f3800e960 tx=0x7f6f3800ee20 comp rx=0 tx=0).stop 2024-09-13T16:44:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.231+0000 7f6f4d800640 1 -- 172.21.15.28:0/2984640521 shutdown_connections 2024-09-13T16:44:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.231+0000 7f6f4d800640 1 --2- 172.21.15.28:0/2984640521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6f2806ac30 0x7f6f2806d0f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.231+0000 7f6f4d800640 1 --2- 172.21.15.28:0/2984640521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6f48150b30 0x7f6f48153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.231+0000 7f6f4d800640 1 --2- 172.21.15.28:0/2984640521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f480cafb0 0x7f6f48151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.231+0000 7f6f4d800640 1 --2- 172.21.15.28:0/2984640521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6f480c9970 0x7f6f48149640 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.231+0000 7f6f4d800640 1 -- 172.21.15.28:0/2984640521 >> 172.21.15.28:0/2984640521 conn(0x7f6f480ca2f0 msgr2=0x7f6f481550a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.231+0000 7f6f4d800640 1 -- 172.21.15.28:0/2984640521 shutdown_connections 2024-09-13T16:44:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.231+0000 7f6f4d800640 1 -- 172.21.15.28:0/2984640521 wait complete. 2024-09-13T16:44:37.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:44:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.404+0000 7f1b93000640 1 Processor -- start 2024-09-13T16:44:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.404+0000 7f1b93000640 1 -- start start 2024-09-13T16:44:37.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.406+0000 7f1b93000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1b8c150740 0x7f1b8c0c44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:37.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.406+0000 7f1b93000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c159e20 con 0x7f1b8c15ab50 2024-09-13T16:44:37.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.406+0000 7f1b93000640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c15a6c0 con 0x7f1b8c0c2170 2024-09-13T16:44:37.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.406+0000 7f1b93000640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b8c0c2790 con 0x7f1b8c150740 2024-09-13T16:44:37.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.406+0000 7f1b91200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1b8c15ab50 0x7f1b8c156830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53788/0 (socket says 172.21.15.28:53788) 2024-09-13T16:44:37.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.406+0000 7f1b91200640 1 -- 172.21.15.28:0/2859394476 learned_addr learned my addr 172.21.15.28:0/2859394476 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:37.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.406+0000 7f1b91c00640 1 --2- 172.21.15.28:0/2859394476 >> v2:172.21.15.191:3300/0 conn(0x7f1b8c150740 0x7f1b8c0c44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:37.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.406+0000 7f1b8be00640 1 -- 172.21.15.28:0/2859394476 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344998361 0 0) 0x7f1b8c159e20 con 0x7f1b8c15ab50 2024-09-13T16:44:37.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.406+0000 7f1b8be00640 1 -- 172.21.15.28:0/2859394476 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b74003520 con 0x7f1b8c15ab50 2024-09-13T16:44:37.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.406+0000 7f1b8be00640 1 -- 172.21.15.28:0/2859394476 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556233304 0 0) 0x7f1b8c15a6c0 con 0x7f1b8c0c2170 2024-09-13T16:44:37.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.406+0000 7f1b8be00640 1 -- 172.21.15.28:0/2859394476 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b8c159e20 con 0x7f1b8c0c2170 2024-09-13T16:44:37.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.407+0000 7f1b91c00640 1 -- 172.21.15.28:0/2859394476 >> v1:172.21.15.191:6789/0 conn(0x7f1b8c0c2170 legacy=0x7f1b8c0c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.407+0000 7f1b91c00640 1 -- 172.21.15.28:0/2859394476 >> v1:172.21.15.28:6789/0 conn(0x7f1b8c15ab50 legacy=0x7f1b8c156830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.407+0000 7f1b91c00640 1 -- 172.21.15.28:0/2859394476 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c0c2a50 con 0x7f1b8c150740 2024-09-13T16:44:37.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.408+0000 7f1b91c00640 1 --2- 172.21.15.28:0/2859394476 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b8c150740 0x7f1b8c0c44a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1b800097a0 tx=0x7f1b8002a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=664df9d7f965719f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:37.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.408+0000 7f1b8be00640 1 -- 172.21.15.28:0/2859394476 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b80032640 con 0x7f1b8c150740 2024-09-13T16:44:37.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.408+0000 7f1b8be00640 1 -- 172.21.15.28:0/2859394476 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b800327a0 con 0x7f1b8c150740 2024-09-13T16:44:37.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.408+0000 7f1b8be00640 1 -- 172.21.15.28:0/2859394476 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b80032b30 con 0x7f1b8c150740 2024-09-13T16:44:37.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.408+0000 7f1b93000640 1 -- 172.21.15.28:0/2859394476 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b8c150740 msgr2=0x7f1b8c0c44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.408+0000 7f1b93000640 1 --2- 172.21.15.28:0/2859394476 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b8c150740 0x7f1b8c0c44a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1b800097a0 tx=0x7f1b8002a2c0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.409+0000 7f1b93000640 1 -- 172.21.15.28:0/2859394476 shutdown_connections 2024-09-13T16:44:37.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.409+0000 7f1b93000640 1 --2- 172.21.15.28:0/2859394476 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b8c150740 0x7f1b8c0c44a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.409+0000 7f1b93000640 1 -- 172.21.15.28:0/2859394476 >> 172.21.15.28:0/2859394476 conn(0x7f1b8c069e80 msgr2=0x7f1b8c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:37.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.409+0000 7f1b93000640 1 -- 172.21.15.28:0/2859394476 shutdown_connections 2024-09-13T16:44:37.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.409+0000 7f1b93000640 1 -- 172.21.15.28:0/2859394476 wait complete. 2024-09-13T16:44:37.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.410+0000 7f1b93000640 1 Processor -- start 2024-09-13T16:44:37.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.410+0000 7f1b93000640 1 -- start start 2024-09-13T16:44:37.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.411+0000 7f1b93000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b8c0c2170 0x7f1b8c1f01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:37.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.411+0000 7f1b93000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8c150740 0x7f1b8c0ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:37.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.411+0000 7f1b93000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b8c15ab50 0x7f1b8c1eca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:37.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.411+0000 7f1b93000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c1529c0 con 0x7f1b8c150740 2024-09-13T16:44:37.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.411+0000 7f1b93000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c1ec590 con 0x7f1b8c15ab50 2024-09-13T16:44:37.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.411+0000 7f1b93000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c151120 con 0x7f1b8c0c2170 2024-09-13T16:44:37.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.411+0000 7f1b91200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8c150740 0x7f1b8c0ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:37.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.412+0000 7f1b91200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8c150740 0x7f1b8c0ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44298/0 (socket says 172.21.15.28:44298) 2024-09-13T16:44:37.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.412+0000 7f1b91200640 1 -- 172.21.15.28:0/3837652659 learned_addr learned my addr 172.21.15.28:0/3837652659 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:37.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.412+0000 7f1b92600640 1 --2- 172.21.15.28:0/3837652659 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b8c15ab50 0x7f1b8c1eca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:37.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.412+0000 7f1b91c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b8c0c2170 0x7f1b8c1f01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:37.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.412+0000 7f1b91200640 1 -- 172.21.15.28:0/3837652659 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b8c0c2170 msgr2=0x7f1b8c1f01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.412+0000 7f1b91200640 1 --2- 172.21.15.28:0/3837652659 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b8c0c2170 0x7f1b8c1f01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.412+0000 7f1b91200640 1 -- 172.21.15.28:0/3837652659 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b8c15ab50 msgr2=0x7f1b8c1eca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.412+0000 7f1b91200640 1 --2- 172.21.15.28:0/3837652659 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b8c15ab50 0x7f1b8c1eca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.412+0000 7f1b91200640 1 -- 172.21.15.28:0/3837652659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b80009400 con 0x7f1b8c150740 2024-09-13T16:44:37.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.412+0000 7f1b92600640 1 --2- 172.21.15.28:0/3837652659 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b8c15ab50 0x7f1b8c1eca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:37.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.412+0000 7f1b91200640 1 --2- 172.21.15.28:0/3837652659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8c150740 0x7f1b8c0ca540 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800c500 tx=0x7f1b7800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:37.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.413+0000 7f1b89600640 1 -- 172.21.15.28:0/3837652659 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b78005ce0 con 0x7f1b8c150740 2024-09-13T16:44:37.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.413+0000 7f1b89600640 1 -- 172.21.15.28:0/3837652659 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b78005e60 con 0x7f1b8c150740 2024-09-13T16:44:37.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.413+0000 7f1b89600640 1 -- 172.21.15.28:0/3837652659 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b78007330 con 0x7f1b8c150740 2024-09-13T16:44:37.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.413+0000 7f1b93000640 1 -- 172.21.15.28:0/3837652659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8c1ec240 con 0x7f1b8c150740 2024-09-13T16:44:37.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.413+0000 7f1b93000640 1 -- 172.21.15.28:0/3837652659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8c153b40 con 0x7f1b8c150740 2024-09-13T16:44:37.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.414+0000 7f1b93000640 1 -- 172.21.15.28:0/3837652659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8c1ec700 con 0x7f1b8c150740 2024-09-13T16:44:37.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.415+0000 7f1b89600640 1 -- 172.21.15.28:0/3837652659 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1b78007550 con 0x7f1b8c150740 2024-09-13T16:44:37.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.416+0000 7f1b89600640 1 --2- 172.21.15.28:0/3837652659 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1b6c06ac00 0x7f1b6c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:37.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.416+0000 7f1b89600640 1 -- 172.21.15.28:0/3837652659 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b78088960 con 0x7f1b8c150740 2024-09-13T16:44:37.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.417+0000 7f1b91c00640 1 --2- 172.21.15.28:0/3837652659 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1b6c06ac00 0x7f1b6c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:37.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.417+0000 7f1b91c00640 1 --2- 172.21.15.28:0/3837652659 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1b6c06ac00 0x7f1b6c06d0c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1b80034000 tx=0x7f1b8002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:37.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.420+0000 7f1b89600640 1 -- 172.21.15.28:0/3837652659 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1b78053ca0 con 0x7f1b8c150740 2024-09-13T16:44:37.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.602+0000 7f1b93000640 1 -- 172.21.15.28:0/3837652659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1b8c0c44a0 con 0x7f1b8c150740 2024-09-13T16:44:37.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.602+0000 7f1b89600640 1 -- 172.21.15.28:0/3837652659 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1b780170a0 con 0x7f1b8c150740 2024-09-13T16:44:37.604 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T16:44:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.605+0000 7f1b93000640 1 -- 172.21.15.28:0/3837652659 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1b6c06ac00 msgr2=0x7f1b6c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.605+0000 7f1b93000640 1 --2- 172.21.15.28:0/3837652659 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1b6c06ac00 0x7f1b6c06d0c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1b80034000 tx=0x7f1b8002e000 comp rx=0 tx=0).stop 2024-09-13T16:44:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.605+0000 7f1b93000640 1 -- 172.21.15.28:0/3837652659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8c150740 msgr2=0x7f1b8c0ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.605+0000 7f1b93000640 1 --2- 172.21.15.28:0/3837652659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8c150740 0x7f1b8c0ca540 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800c500 tx=0x7f1b7800c9c0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.605+0000 7f1b93000640 1 -- 172.21.15.28:0/3837652659 shutdown_connections 2024-09-13T16:44:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.605+0000 7f1b93000640 1 --2- 172.21.15.28:0/3837652659 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1b6c06ac00 0x7f1b6c06d0c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.605+0000 7f1b93000640 1 --2- 172.21.15.28:0/3837652659 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1b8c15ab50 0x7f1b8c1eca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.605+0000 7f1b93000640 1 --2- 172.21.15.28:0/3837652659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8c150740 0x7f1b8c0ca540 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.605+0000 7f1b93000640 1 --2- 172.21.15.28:0/3837652659 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1b8c0c2170 0x7f1b8c1f01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.605+0000 7f1b93000640 1 -- 172.21.15.28:0/3837652659 >> 172.21.15.28:0/3837652659 conn(0x7f1b8c069e80 msgr2=0x7f1b8c0c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:37.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.605+0000 7f1b93000640 1 -- 172.21.15.28:0/3837652659 shutdown_connections 2024-09-13T16:44:37.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.605+0000 7f1b93000640 1 -- 172.21.15.28:0/3837652659 wait complete. 2024-09-13T16:44:37.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:44:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.781+0000 7f0043800640 1 Processor -- start 2024-09-13T16:44:37.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.781+0000 7f0043800640 1 -- start start 2024-09-13T16:44:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.782+0000 7f0043800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f003c0c5140 0x7f003c159800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.782+0000 7f0043800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f003c0c73e0 0x7f003c0c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.782+0000 7f0043800640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c0c6f50 con 0x7f003c0c5a60 2024-09-13T16:44:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.782+0000 7f0043800640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f003c0c6940 con 0x7f003c0c73e0 2024-09-13T16:44:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.782+0000 7f0043800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003c159d40 con 0x7f003c0c5140 2024-09-13T16:44:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.783+0000 7f0040e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f003c0c5140 0x7f003c159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.783+0000 7f003be00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f003c0c73e0 0x7f003c0c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.783+0000 7f0040e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f003c0c5140 0x7f003c159800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:55438/0 (socket says 172.21.15.28:55438) 2024-09-13T16:44:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.783+0000 7f0041800640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f003c0c5a60 0x7f003c14f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:42766/0 (socket says 172.21.15.28:42766) 2024-09-13T16:44:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.783+0000 7f0040e00640 1 -- 172.21.15.28:0/606590527 learned_addr learned my addr 172.21.15.28:0/606590527 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.783+0000 7f003be00640 1 -- 172.21.15.28:0/606590527 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f003c0c5140 msgr2=0x7f003c159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.783+0000 7f003be00640 1 --2- 172.21.15.28:0/606590527 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f003c0c5140 0x7f003c159800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.783+0000 7f003be00640 1 -- 172.21.15.28:0/606590527 >> v1:172.21.15.191:6789/0 conn(0x7f003c0c5a60 legacy=0x7f003c14f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.783+0000 7f003be00640 1 -- 172.21.15.28:0/606590527 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003c0c1b50 con 0x7f003c0c73e0 2024-09-13T16:44:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.783+0000 7f003b400640 1 -- 172.21.15.28:0/606590527 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967427486 0 0) 0x7f003c0c6f50 con 0x7f003c0c5a60 2024-09-13T16:44:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.783+0000 7f0040e00640 1 --2- 172.21.15.28:0/606590527 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f003c0c5140 0x7f003c159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.784+0000 7f003be00640 1 --2- 172.21.15.28:0/606590527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f003c0c73e0 0x7f003c0c5520 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f00280097a0 tx=0x7f002802a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a976a724b3d6b911 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.784+0000 7f003b400640 1 -- 172.21.15.28:0/606590527 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002802ae50 con 0x7f003c0c73e0 2024-09-13T16:44:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.784+0000 7f003b400640 1 -- 172.21.15.28:0/606590527 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0028032440 con 0x7f003c0c73e0 2024-09-13T16:44:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.784+0000 7f003b400640 1 -- 172.21.15.28:0/606590527 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00280327d0 con 0x7f003c0c73e0 2024-09-13T16:44:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.785+0000 7f0043800640 1 -- 172.21.15.28:0/606590527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f003c0c73e0 msgr2=0x7f003c0c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.785+0000 7f0043800640 1 --2- 172.21.15.28:0/606590527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f003c0c73e0 0x7f003c0c5520 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f00280097a0 tx=0x7f002802a100 comp rx=0 tx=0).stop 2024-09-13T16:44:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.785+0000 7f0043800640 1 -- 172.21.15.28:0/606590527 shutdown_connections 2024-09-13T16:44:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.785+0000 7f0043800640 1 --2- 172.21.15.28:0/606590527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f003c0c73e0 0x7f003c0c5520 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.785+0000 7f0043800640 1 --2- 172.21.15.28:0/606590527 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f003c0c5140 0x7f003c159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.785+0000 7f0043800640 1 -- 172.21.15.28:0/606590527 >> 172.21.15.28:0/606590527 conn(0x7f003c0ca6b0 msgr2=0x7f003c0caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.785+0000 7f0043800640 1 -- 172.21.15.28:0/606590527 shutdown_connections 2024-09-13T16:44:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.785+0000 7f0043800640 1 -- 172.21.15.28:0/606590527 wait complete. 2024-09-13T16:44:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.787+0000 7f0043800640 1 Processor -- start 2024-09-13T16:44:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.787+0000 7f0043800640 1 -- start start 2024-09-13T16:44:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.787+0000 7f0043800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f003c0c5140 0x7f003c1f12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.787+0000 7f0043800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f003c0c5a60 0x7f003c160b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.788+0000 7f0043800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f003c0c73e0 0x7f003c0c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.788+0000 7f0043800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003c15ed00 con 0x7f003c0c5140 2024-09-13T16:44:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.788+0000 7f0043800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003c14dcf0 con 0x7f003c0c5a60 2024-09-13T16:44:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.788+0000 7f0043800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003c160710 con 0x7f003c0c73e0 2024-09-13T16:44:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.788+0000 7f0040e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f003c0c5140 0x7f003c1f12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.788+0000 7f0041800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f003c0c73e0 0x7f003c0c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.788+0000 7f003be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f003c0c5a60 0x7f003c160b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.788+0000 7f0040e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f003c0c5140 0x7f003c1f12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44310/0 (socket says 172.21.15.28:44310) 2024-09-13T16:44:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.788+0000 7f0040e00640 1 -- 172.21.15.28:0/3948545290 learned_addr learned my addr 172.21.15.28:0/3948545290 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.788+0000 7f003be00640 1 --2- 172.21.15.28:0/3948545290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f003c0c5a60 0x7f003c160b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:50884/0 (socket says 172.21.15.28:50884) 2024-09-13T16:44:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.788+0000 7f003be00640 1 -- 172.21.15.28:0/3948545290 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f003c0c73e0 msgr2=0x7f003c0c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.788+0000 7f003be00640 1 --2- 172.21.15.28:0/3948545290 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f003c0c73e0 0x7f003c0c97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.788+0000 7f003be00640 1 -- 172.21.15.28:0/3948545290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f003c0c5140 msgr2=0x7f003c1f12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.788+0000 7f003be00640 1 --2- 172.21.15.28:0/3948545290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f003c0c5140 0x7f003c1f12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.788+0000 7f003be00640 1 -- 172.21.15.28:0/3948545290 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00300094d0 con 0x7f003c0c5a60 2024-09-13T16:44:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.789+0000 7f003be00640 1 --2- 172.21.15.28:0/3948545290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f003c0c5a60 0x7f003c160b40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0028031000 tx=0x7f00280046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.789+0000 7f0038c00640 1 -- 172.21.15.28:0/3948545290 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0028004230 con 0x7f003c0c5a60 2024-09-13T16:44:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.789+0000 7f0038c00640 1 -- 172.21.15.28:0/3948545290 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0028004390 con 0x7f003c0c5a60 2024-09-13T16:44:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.789+0000 7f0038c00640 1 -- 172.21.15.28:0/3948545290 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0028003850 con 0x7f003c0c5a60 2024-09-13T16:44:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.789+0000 7f0043800640 1 -- 172.21.15.28:0/3948545290 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0028009400 con 0x7f003c0c5a60 2024-09-13T16:44:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.789+0000 7f0043800640 1 -- 172.21.15.28:0/3948545290 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f003c15f1f0 con 0x7f003c0c5a60 2024-09-13T16:44:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.790+0000 7f0043800640 1 -- 172.21.15.28:0/3948545290 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f003c06fe80 con 0x7f003c0c5a60 2024-09-13T16:44:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.791+0000 7f0038c00640 1 -- 172.21.15.28:0/3948545290 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0028039050 con 0x7f003c0c5a60 2024-09-13T16:44:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.792+0000 7f0038c00640 1 --2- 172.21.15.28:0/3948545290 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f001c06ac00 0x7f001c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.792+0000 7f0038c00640 1 -- 172.21.15.28:0/3948545290 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00280a99b0 con 0x7f003c0c5a60 2024-09-13T16:44:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.792+0000 7f0040e00640 1 --2- 172.21.15.28:0/3948545290 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f001c06ac00 0x7f001c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.793+0000 7f0040e00640 1 --2- 172.21.15.28:0/3948545290 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f001c06ac00 0x7f001c06d0c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0030009e20 tx=0x7f0030009240 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:37.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.797+0000 7f0038c00640 1 -- 172.21.15.28:0/3948545290 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0028074b10 con 0x7f003c0c5a60 2024-09-13T16:44:37.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.976+0000 7f0043800640 1 -- 172.21.15.28:0/3948545290 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f003c15ef10 con 0x7f003c0c5a60 2024-09-13T16:44:37.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.977+0000 7f0038c00640 1 -- 172.21.15.28:0/3948545290 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f002804b020 con 0x7f003c0c5a60 2024-09-13T16:44:37.979 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T16:44:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.979+0000 7f0043800640 1 -- 172.21.15.28:0/3948545290 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f001c06ac00 msgr2=0x7f001c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.979+0000 7f0043800640 1 --2- 172.21.15.28:0/3948545290 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f001c06ac00 0x7f001c06d0c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0030009e20 tx=0x7f0030009240 comp rx=0 tx=0).stop 2024-09-13T16:44:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.979+0000 7f0043800640 1 -- 172.21.15.28:0/3948545290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f003c0c5a60 msgr2=0x7f003c160b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.979+0000 7f0043800640 1 --2- 172.21.15.28:0/3948545290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f003c0c5a60 0x7f003c160b40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0028031000 tx=0x7f00280046c0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.979+0000 7f0043800640 1 -- 172.21.15.28:0/3948545290 shutdown_connections 2024-09-13T16:44:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.979+0000 7f0043800640 1 --2- 172.21.15.28:0/3948545290 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f001c06ac00 0x7f001c06d0c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.979+0000 7f0043800640 1 --2- 172.21.15.28:0/3948545290 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f003c0c73e0 0x7f003c0c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.979+0000 7f0043800640 1 --2- 172.21.15.28:0/3948545290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f003c0c5a60 0x7f003c160b40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.979+0000 7f0043800640 1 --2- 172.21.15.28:0/3948545290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f003c0c5140 0x7f003c1f12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:37.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.979+0000 7f0043800640 1 -- 172.21.15.28:0/3948545290 >> 172.21.15.28:0/3948545290 conn(0x7f003c0ca6b0 msgr2=0x7f003c13d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:37.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.979+0000 7f0043800640 1 -- 172.21.15.28:0/3948545290 shutdown_connections 2024-09-13T16:44:37.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:37.979+0000 7f0043800640 1 -- 172.21.15.28:0/3948545290 wait complete. 2024-09-13T16:44:37.996 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-13T16:44:37.996 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:44:38.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.149+0000 7f24e7200640 1 Processor -- start 2024-09-13T16:44:38.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.149+0000 7f24e7200640 1 -- start start 2024-09-13T16:44:38.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.150+0000 7f24e7200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f24e0152f20 0x7f24e015c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:38.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.150+0000 7f24e7200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24e00c6a00 0x7f24e015cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:38.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.150+0000 7f24e7200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e015c340 con 0x7f24e0158b20 2024-09-13T16:44:38.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.150+0000 7f24e7200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24e0159b50 con 0x7f24e00c6a00 2024-09-13T16:44:38.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.150+0000 7f24e7200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e0158fd0 con 0x7f24e0152f20 2024-09-13T16:44:38.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.151+0000 7f24e5200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24e00c6a00 0x7f24e015cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:38.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.151+0000 7f24dfe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f24e0152f20 0x7f24e015c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:38.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.151+0000 7f24d7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f24e0158b20 0x7f24e00c5ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53794/0 (socket says 172.21.15.28:53794) 2024-09-13T16:44:38.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.151+0000 7f24d7e00640 1 -- 172.21.15.28:0/2331469704 learned_addr learned my addr 172.21.15.28:0/2331469704 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:38.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.151+0000 7f24dfe00640 1 -- 172.21.15.28:0/2331469704 >> v2:172.21.15.28:3300/0 conn(0x7f24e00c6a00 msgr2=0x7f24e015cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:38.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.151+0000 7f24dfe00640 1 --2- 172.21.15.28:0/2331469704 >> v2:172.21.15.28:3300/0 conn(0x7f24e00c6a00 0x7f24e015cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.151+0000 7f24dfe00640 1 -- 172.21.15.28:0/2331469704 >> v1:172.21.15.28:6789/0 conn(0x7f24e0158b20 legacy=0x7f24e00c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.151+0000 7f24dfe00640 1 -- 172.21.15.28:0/2331469704 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e01592b0 con 0x7f24e0152f20 2024-09-13T16:44:38.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.151+0000 7f24e5200640 1 --2- 172.21.15.28:0/2331469704 >> v2:172.21.15.28:3300/0 conn(0x7f24e00c6a00 0x7f24e015cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:38.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.152+0000 7f24dfe00640 1 --2- 172.21.15.28:0/2331469704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f24e0152f20 0x7f24e015c7d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f24d00097a0 tx=0x7f24d002a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=491d520ca3776eb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:38.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.153+0000 7f24df400640 1 -- 172.21.15.28:0/2331469704 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d0032440 con 0x7f24e0152f20 2024-09-13T16:44:38.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.153+0000 7f24df400640 1 -- 172.21.15.28:0/2331469704 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d00325a0 con 0x7f24e0152f20 2024-09-13T16:44:38.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.153+0000 7f24df400640 1 -- 172.21.15.28:0/2331469704 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d0032930 con 0x7f24e0152f20 2024-09-13T16:44:38.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.153+0000 7f24e7200640 1 -- 172.21.15.28:0/2331469704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f24e0152f20 msgr2=0x7f24e015c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.153+0000 7f24e7200640 1 --2- 172.21.15.28:0/2331469704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f24e0152f20 0x7f24e015c7d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f24d00097a0 tx=0x7f24d002a230 comp rx=0 tx=0).stop 2024-09-13T16:44:38.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.153+0000 7f24e7200640 1 -- 172.21.15.28:0/2331469704 shutdown_connections 2024-09-13T16:44:38.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.153+0000 7f24e7200640 1 --2- 172.21.15.28:0/2331469704 >> v2:172.21.15.28:3300/0 conn(0x7f24e00c6a00 0x7f24e015cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.153+0000 7f24e7200640 1 --2- 172.21.15.28:0/2331469704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f24e0152f20 0x7f24e015c7d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.153+0000 7f24e7200640 1 -- 172.21.15.28:0/2331469704 >> 172.21.15.28:0/2331469704 conn(0x7f24e0066040 msgr2=0x7f24e0069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:38.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.153+0000 7f24e7200640 1 -- 172.21.15.28:0/2331469704 shutdown_connections 2024-09-13T16:44:38.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.153+0000 7f24e7200640 1 -- 172.21.15.28:0/2331469704 wait complete. 2024-09-13T16:44:38.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.155+0000 7f24e7200640 1 Processor -- start 2024-09-13T16:44:38.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.155+0000 7f24e7200640 1 -- start start 2024-09-13T16:44:38.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.156+0000 7f24e7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e00c6a00 0x7f24e0154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:38.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.156+0000 7f24e7200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24e0152f20 0x7f24e01f1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:38.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.156+0000 7f24e7200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f24e0158b20 0x7f24e0157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:38.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.156+0000 7f24e7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e0155040 con 0x7f24e00c6a00 2024-09-13T16:44:38.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.156+0000 7f24e7200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e014d130 con 0x7f24e0152f20 2024-09-13T16:44:38.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.156+0000 7f24e7200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e014d6f0 con 0x7f24e0158b20 2024-09-13T16:44:38.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.156+0000 7f24dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e00c6a00 0x7f24e0154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:38.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.156+0000 7f24d7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24e0152f20 0x7f24e01f1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:38.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.156+0000 7f24dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e00c6a00 0x7f24e0154b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44322/0 (socket says 172.21.15.28:44322) 2024-09-13T16:44:38.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.156+0000 7f24dfe00640 1 -- 172.21.15.28:0/3317430419 learned_addr learned my addr 172.21.15.28:0/3317430419 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:38.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.156+0000 7f24e5200640 1 --2- 172.21.15.28:0/3317430419 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f24e0158b20 0x7f24e0157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:38.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.157+0000 7f24dfe00640 1 -- 172.21.15.28:0/3317430419 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f24e0158b20 msgr2=0x7f24e0157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.157+0000 7f24dfe00640 1 --2- 172.21.15.28:0/3317430419 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f24e0158b20 0x7f24e0157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.157+0000 7f24dfe00640 1 -- 172.21.15.28:0/3317430419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24e0152f20 msgr2=0x7f24e01f1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.157+0000 7f24dfe00640 1 --2- 172.21.15.28:0/3317430419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24e0152f20 0x7f24e01f1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.157+0000 7f24dfe00640 1 -- 172.21.15.28:0/3317430419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d8009a70 con 0x7f24e00c6a00 2024-09-13T16:44:38.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.157+0000 7f24e5200640 1 --2- 172.21.15.28:0/3317430419 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f24e0158b20 0x7f24e0157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:38.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.157+0000 7f24dfe00640 1 --2- 172.21.15.28:0/3317430419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e00c6a00 0x7f24e0154b00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f24d00098c0 tx=0x7f24d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:38.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.157+0000 7f24dcc00640 1 -- 172.21.15.28:0/3317430419 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d0032440 con 0x7f24e00c6a00 2024-09-13T16:44:38.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.157+0000 7f24dcc00640 1 -- 172.21.15.28:0/3317430419 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d0032e90 con 0x7f24e00c6a00 2024-09-13T16:44:38.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.157+0000 7f24dcc00640 1 -- 172.21.15.28:0/3317430419 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d0030ba0 con 0x7f24e00c6a00 2024-09-13T16:44:38.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.157+0000 7f24e7200640 1 -- 172.21.15.28:0/3317430419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d0009400 con 0x7f24e00c6a00 2024-09-13T16:44:38.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.157+0000 7f24e7200640 1 -- 172.21.15.28:0/3317430419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e01f21c0 con 0x7f24e00c6a00 2024-09-13T16:44:38.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.159+0000 7f24e7200640 1 -- 172.21.15.28:0/3317430419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e00c5180 con 0x7f24e00c6a00 2024-09-13T16:44:38.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.159+0000 7f24dcc00640 1 -- 172.21.15.28:0/3317430419 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f24d0039040 con 0x7f24e00c6a00 2024-09-13T16:44:38.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.160+0000 7f24dcc00640 1 --2- 172.21.15.28:0/3317430419 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f24bc06ac00 0x7f24bc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:38.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.160+0000 7f24dcc00640 1 -- 172.21.15.28:0/3317430419 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24d00a9b00 con 0x7f24e00c6a00 2024-09-13T16:44:38.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.161+0000 7f24d7e00640 1 --2- 172.21.15.28:0/3317430419 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f24bc06ac00 0x7f24bc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:38.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.161+0000 7f24d7e00640 1 --2- 172.21.15.28:0/3317430419 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f24bc06ac00 0x7f24bc06d0c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f24e00c7820 tx=0x7f24c8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.165+0000 7f24dcc00640 1 -- 172.21.15.28:0/3317430419 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f24d00abe60 con 0x7f24e00c6a00 2024-09-13T16:44:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.339+0000 7f24e7200640 1 -- 172.21.15.28:0/3317430419 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f24e0159870 con 0x7f24bc06ac00 2024-09-13T16:44:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.347+0000 7f24dcc00640 1 -- 172.21.15.28:0/3317430419 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264747 (secure 0 0 0) 0x7f24c80014e0 con 0x7f24bc06ac00 2024-09-13T16:44:38.349 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:38.359 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T16:44:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.351+0000 7f24e7200640 1 -- 172.21.15.28:0/3317430419 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f24bc06ac00 msgr2=0x7f24bc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.351+0000 7f24e7200640 1 --2- 172.21.15.28:0/3317430419 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f24bc06ac00 0x7f24bc06d0c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f24e00c7820 tx=0x7f24c8009040 comp rx=0 tx=0).stop 2024-09-13T16:44:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.351+0000 7f24e7200640 1 -- 172.21.15.28:0/3317430419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e00c6a00 msgr2=0x7f24e0154b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.351+0000 7f24e7200640 1 --2- 172.21.15.28:0/3317430419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e00c6a00 0x7f24e0154b00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f24d00098c0 tx=0x7f24d00046c0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.351+0000 7f24e7200640 1 -- 172.21.15.28:0/3317430419 shutdown_connections 2024-09-13T16:44:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.351+0000 7f24e7200640 1 --2- 172.21.15.28:0/3317430419 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f24bc06ac00 0x7f24bc06d0c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.351+0000 7f24e7200640 1 --2- 172.21.15.28:0/3317430419 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f24e0158b20 0x7f24e0157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.351+0000 7f24e7200640 1 --2- 172.21.15.28:0/3317430419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24e0152f20 0x7f24e01f1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.351+0000 7f24e7200640 1 --2- 172.21.15.28:0/3317430419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e00c6a00 0x7f24e0154b00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.351+0000 7f24e7200640 1 -- 172.21.15.28:0/3317430419 >> 172.21.15.28:0/3317430419 conn(0x7f24e0066040 msgr2=0x7f24e00702b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.351+0000 7f24e7200640 1 -- 172.21.15.28:0/3317430419 shutdown_connections 2024-09-13T16:44:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.351+0000 7f24e7200640 1 -- 172.21.15.28:0/3317430419 wait complete. 2024-09-13T16:44:38.368 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-09-13T16:44:37.601103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598162,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":364,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222196,"kb_used_data":1972,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754752524,"statfs":{"total":773094113280,"available":772866584576,"internally_reserved":0,"allocated":2019328,"data_stored":1761261,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12743,"internal_metadata":225431097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1358,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002968"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178474+0000","last_change":"2024-09-13T16:43:54.987241+0000","last_active":"2024-09-13T16:43:59.178474+0000","last_peered":"2024-09-13T16:43:59.178474+0000","last_clean":"2024-09-13T16:43:59.178474+0000","last_became_active":"2024-09-13T16:43:54.986988+0000","last_became_peered":"2024-09-13T16:43:54.986988+0000","last_unstale":"2024-09-13T16:43:59.178474+0000","last_undegraded":"2024-09-13T16:43:59.178474+0000","last_fullsized":"2024-09-13T16:43:59.178474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975685+0000","last_change":"2024-09-13T16:43:55.001141+0000","last_active":"2024-09-13T16:43:55.975685+0000","last_peered":"2024-09-13T16:43:55.975685+0000","last_clean":"2024-09-13T16:43:55.975685+0000","last_became_active":"2024-09-13T16:43:55.000757+0000","last_became_peered":"2024-09-13T16:43:55.000757+0000","last_unstale":"2024-09-13T16:43:55.975685+0000","last_undegraded":"2024-09-13T16:43:55.975685+0000","last_fullsized":"2024-09-13T16:43:55.975685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872108+0000","last_change":"2024-09-13T16:43:54.986925+0000","last_active":"2024-09-13T16:43:56.872108+0000","last_peered":"2024-09-13T16:43:56.872108+0000","last_clean":"2024-09-13T16:43:56.872108+0000","last_became_active":"2024-09-13T16:43:54.986198+0000","last_became_peered":"2024-09-13T16:43:54.986198+0000","last_unstale":"2024-09-13T16:43:56.872108+0000","last_undegraded":"2024-09-13T16:43:56.872108+0000","last_fullsized":"2024-09-13T16:43:56.872108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:03.127550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485981+0000","last_change":"2024-09-13T16:43:54.992249+0000","last_active":"2024-09-13T16:43:55.485981+0000","last_peered":"2024-09-13T16:43:55.485981+0000","last_clean":"2024-09-13T16:43:55.485981+0000","last_became_active":"2024-09-13T16:43:54.991804+0000","last_became_peered":"2024-09-13T16:43:54.991804+0000","last_unstale":"2024-09-13T16:43:55.485981+0000","last_undegraded":"2024-09-13T16:43:55.485981+0000","last_fullsized":"2024-09-13T16:43:55.485981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178163+0000","last_change":"2024-09-13T16:43:54.987564+0000","last_active":"2024-09-13T16:43:59.178163+0000","last_peered":"2024-09-13T16:43:59.178163+0000","last_clean":"2024-09-13T16:43:59.178163+0000","last_became_active":"2024-09-13T16:43:54.987328+0000","last_became_peered":"2024-09-13T16:43:54.987328+0000","last_unstale":"2024-09-13T16:43:59.178163+0000","last_undegraded":"2024-09-13T16:43:59.178163+0000","last_fullsized":"2024-09-13T16:43:59.178163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:36.006730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485919+0000","last_change":"2024-09-13T16:43:55.000019+0000","last_active":"2024-09-13T16:43:55.485919+0000","last_peered":"2024-09-13T16:43:55.485919+0000","last_clean":"2024-09-13T16:43:55.485919+0000","last_became_active":"2024-09-13T16:43:54.999752+0000","last_became_peered":"2024-09-13T16:43:54.999752+0000","last_unstale":"2024-09-13T16:43:55.485919+0000","last_undegraded":"2024-09-13T16:43:55.485919+0000","last_fullsized":"2024-09-13T16:43:55.485919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.177985+0000","last_change":"2024-09-13T16:43:54.999533+0000","last_active":"2024-09-13T16:43:59.177985+0000","last_peered":"2024-09-13T16:43:59.177985+0000","last_clean":"2024-09-13T16:43:59.177985+0000","last_became_active":"2024-09-13T16:43:54.999350+0000","last_became_peered":"2024-09-13T16:43:54.999350+0000","last_unstale":"2024-09-13T16:43:59.177985+0000","last_undegraded":"2024-09-13T16:43:59.177985+0000","last_fullsized":"2024-09-13T16:43:59.177985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:29.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508531+0000","last_change":"2024-09-13T16:43:54.998526+0000","last_active":"2024-09-13T16:43:56.508531+0000","last_peered":"2024-09-13T16:43:56.508531+0000","last_clean":"2024-09-13T16:43:56.508531+0000","last_became_active":"2024-09-13T16:43:54.996309+0000","last_became_peered":"2024-09-13T16:43:54.996309+0000","last_unstale":"2024-09-13T16:43:56.508531+0000","last_undegraded":"2024-09-13T16:43:56.508531+0000","last_fullsized":"2024-09-13T16:43:56.508531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178473+0000","last_change":"2024-09-13T16:43:54.993786+0000","last_active":"2024-09-13T16:43:59.178473+0000","last_peered":"2024-09-13T16:43:59.178473+0000","last_clean":"2024-09-13T16:43:59.178473+0000","last_became_active":"2024-09-13T16:43:54.993475+0000","last_became_peered":"2024-09-13T16:43:54.993475+0000","last_unstale":"2024-09-13T16:43:59.178473+0000","last_undegraded":"2024-09-13T16:43:59.178473+0000","last_fullsized":"2024-09-13T16:43:59.178473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260988+0000","last_change":"2024-09-13T16:43:54.986272+0000","last_active":"2024-09-13T16:43:56.260988+0000","last_peered":"2024-09-13T16:43:56.260988+0000","last_clean":"2024-09-13T16:43:56.260988+0000","last_became_active":"2024-09-13T16:43:54.985979+0000","last_became_peered":"2024-09-13T16:43:54.985979+0000","last_unstale":"2024-09-13T16:43:56.260988+0000","last_undegraded":"2024-09-13T16:43:56.260988+0000","last_fullsized":"2024-09-13T16:43:56.260988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260766+0000","last_change":"2024-09-13T16:43:54.986207+0000","last_active":"2024-09-13T16:43:56.260766+0000","last_peered":"2024-09-13T16:43:56.260766+0000","last_clean":"2024-09-13T16:43:56.260766+0000","last_became_active":"2024-09-13T16:43:54.985948+0000","last_became_peered":"2024-09-13T16:43:54.985948+0000","last_unstale":"2024-09-13T16:43:56.260766+0000","last_undegraded":"2024-09-13T16:43:56.260766+0000","last_fullsized":"2024-09-13T16:43:56.260766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935591+0000","last_change":"2024-09-13T16:43:54.986636+0000","last_active":"2024-09-13T16:43:56.935591+0000","last_peered":"2024-09-13T16:43:56.935591+0000","last_clean":"2024-09-13T16:43:56.935591+0000","last_became_active":"2024-09-13T16:43:54.986161+0000","last_became_peered":"2024-09-13T16:43:54.986161+0000","last_unstale":"2024-09-13T16:43:56.935591+0000","last_undegraded":"2024-09-13T16:43:56.935591+0000","last_fullsized":"2024-09-13T16:43:56.935591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:09.340330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508720+0000","last_change":"2024-09-13T16:43:54.997783+0000","last_active":"2024-09-13T16:43:56.508720+0000","last_peered":"2024-09-13T16:43:56.508720+0000","last_clean":"2024-09-13T16:43:56.508720+0000","last_became_active":"2024-09-13T16:43:54.997162+0000","last_became_peered":"2024-09-13T16:43:54.997162+0000","last_unstale":"2024-09-13T16:43:56.508720+0000","last_undegraded":"2024-09-13T16:43:56.508720+0000","last_fullsized":"2024-09-13T16:43:56.508720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872216+0000","last_change":"2024-09-13T16:43:54.995262+0000","last_active":"2024-09-13T16:43:56.872216+0000","last_peered":"2024-09-13T16:43:56.872216+0000","last_clean":"2024-09-13T16:43:56.872216+0000","last_became_active":"2024-09-13T16:43:54.994891+0000","last_became_peered":"2024-09-13T16:43:54.994891+0000","last_unstale":"2024-09-13T16:43:56.872216+0000","last_undegraded":"2024-09-13T16:43:56.872216+0000","last_fullsized":"2024-09-13T16:43:56.872216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975286+0000","last_change":"2024-09-13T16:43:54.981534+0000","last_active":"2024-09-13T16:43:55.975286+0000","last_peered":"2024-09-13T16:43:55.975286+0000","last_clean":"2024-09-13T16:43:55.975286+0000","last_became_active":"2024-09-13T16:43:54.981191+0000","last_became_peered":"2024-09-13T16:43:54.981191+0000","last_unstale":"2024-09-13T16:43:55.975286+0000","last_undegraded":"2024-09-13T16:43:55.975286+0000","last_fullsized":"2024-09-13T16:43:55.975286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485880+0000","last_change":"2024-09-13T16:43:54.994157+0000","last_active":"2024-09-13T16:43:55.485880+0000","last_peered":"2024-09-13T16:43:55.485880+0000","last_clean":"2024-09-13T16:43:55.485880+0000","last_became_active":"2024-09-13T16:43:54.993915+0000","last_became_peered":"2024-09-13T16:43:54.993915+0000","last_unstale":"2024-09-13T16:43:55.485880+0000","last_undegraded":"2024-09-13T16:43:55.485880+0000","last_fullsized":"2024-09-13T16:43:55.485880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486053+0000","last_change":"2024-09-13T16:43:54.987328+0000","last_active":"2024-09-13T16:43:55.486053+0000","last_peered":"2024-09-13T16:43:55.486053+0000","last_clean":"2024-09-13T16:43:55.486053+0000","last_became_active":"2024-09-13T16:43:54.987048+0000","last_became_peered":"2024-09-13T16:43:54.987048+0000","last_unstale":"2024-09-13T16:43:55.486053+0000","last_undegraded":"2024-09-13T16:43:55.486053+0000","last_fullsized":"2024-09-13T16:43:55.486053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.979840+0000","last_change":"2024-09-13T16:43:54.985628+0000","last_active":"2024-09-13T16:43:55.979840+0000","last_peered":"2024-09-13T16:43:55.979840+0000","last_clean":"2024-09-13T16:43:55.979840+0000","last_became_active":"2024-09-13T16:43:54.985378+0000","last_became_peered":"2024-09-13T16:43:54.985378+0000","last_unstale":"2024-09-13T16:43:55.979840+0000","last_undegraded":"2024-09-13T16:43:55.979840+0000","last_fullsized":"2024-09-13T16:43:55.979840+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178469+0000","last_change":"2024-09-13T16:43:54.996712+0000","last_active":"2024-09-13T16:43:59.178469+0000","last_peered":"2024-09-13T16:43:59.178469+0000","last_clean":"2024-09-13T16:43:59.178469+0000","last_became_active":"2024-09-13T16:43:54.996523+0000","last_became_peered":"2024-09-13T16:43:54.996523+0000","last_unstale":"2024-09-13T16:43:59.178469+0000","last_undegraded":"2024-09-13T16:43:59.178469+0000","last_fullsized":"2024-09-13T16:43:59.178469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260867+0000","last_change":"2024-09-13T16:43:54.984879+0000","last_active":"2024-09-13T16:43:56.260867+0000","last_peered":"2024-09-13T16:43:56.260867+0000","last_clean":"2024-09-13T16:43:56.260867+0000","last_became_active":"2024-09-13T16:43:54.984595+0000","last_became_peered":"2024-09-13T16:43:54.984595+0000","last_unstale":"2024-09-13T16:43:56.260867+0000","last_undegraded":"2024-09-13T16:43:56.260867+0000","last_fullsized":"2024-09-13T16:43:56.260867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.871914+0000","last_change":"2024-09-13T16:43:54.996457+0000","last_active":"2024-09-13T16:43:56.871914+0000","last_peered":"2024-09-13T16:43:56.871914+0000","last_clean":"2024-09-13T16:43:56.871914+0000","last_became_active":"2024-09-13T16:43:54.996275+0000","last_became_peered":"2024-09-13T16:43:54.996275+0000","last_unstale":"2024-09-13T16:43:56.871914+0000","last_undegraded":"2024-09-13T16:43:56.871914+0000","last_fullsized":"2024-09-13T16:43:56.871914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485929+0000","last_change":"2024-09-13T16:43:54.985368+0000","last_active":"2024-09-13T16:43:55.485929+0000","last_peered":"2024-09-13T16:43:55.485929+0000","last_clean":"2024-09-13T16:43:55.485929+0000","last_became_active":"2024-09-13T16:43:54.985130+0000","last_became_peered":"2024-09-13T16:43:54.985130+0000","last_unstale":"2024-09-13T16:43:55.485929+0000","last_undegraded":"2024-09-13T16:43:55.485929+0000","last_fullsized":"2024-09-13T16:43:55.485929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508531+0000","last_change":"2024-09-13T16:43:54.991862+0000","last_active":"2024-09-13T16:43:56.508531+0000","last_peered":"2024-09-13T16:43:56.508531+0000","last_clean":"2024-09-13T16:43:56.508531+0000","last_became_active":"2024-09-13T16:43:54.991554+0000","last_became_peered":"2024-09-13T16:43:54.991554+0000","last_unstale":"2024-09-13T16:43:56.508531+0000","last_undegraded":"2024-09-13T16:43:56.508531+0000","last_fullsized":"2024-09-13T16:43:56.508531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485896+0000","last_change":"2024-09-13T16:43:54.992241+0000","last_active":"2024-09-13T16:43:55.485896+0000","last_peered":"2024-09-13T16:43:55.485896+0000","last_clean":"2024-09-13T16:43:55.485896+0000","last_became_active":"2024-09-13T16:43:54.991687+0000","last_became_peered":"2024-09-13T16:43:54.991687+0000","last_unstale":"2024-09-13T16:43:55.485896+0000","last_undegraded":"2024-09-13T16:43:55.485896+0000","last_fullsized":"2024-09-13T16:43:55.485896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.509024+0000","last_change":"2024-09-13T16:43:54.991366+0000","last_active":"2024-09-13T16:43:56.509024+0000","last_peered":"2024-09-13T16:43:56.509024+0000","last_clean":"2024-09-13T16:43:56.509024+0000","last_became_active":"2024-09-13T16:43:54.991127+0000","last_became_peered":"2024-09-13T16:43:54.991127+0000","last_unstale":"2024-09-13T16:43:56.509024+0000","last_undegraded":"2024-09-13T16:43:56.509024+0000","last_fullsized":"2024-09-13T16:43:56.509024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.261006+0000","last_change":"2024-09-13T16:43:54.985615+0000","last_active":"2024-09-13T16:43:56.261006+0000","last_peered":"2024-09-13T16:43:56.261006+0000","last_clean":"2024-09-13T16:43:56.261006+0000","last_became_active":"2024-09-13T16:43:54.985299+0000","last_became_peered":"2024-09-13T16:43:54.985299+0000","last_unstale":"2024-09-13T16:43:56.261006+0000","last_undegraded":"2024-09-13T16:43:56.261006+0000","last_fullsized":"2024-09-13T16:43:56.261006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975382+0000","last_change":"2024-09-13T16:43:54.993325+0000","last_active":"2024-09-13T16:43:55.975382+0000","last_peered":"2024-09-13T16:43:55.975382+0000","last_clean":"2024-09-13T16:43:55.975382+0000","last_became_active":"2024-09-13T16:43:54.993009+0000","last_became_peered":"2024-09-13T16:43:54.993009+0000","last_unstale":"2024-09-13T16:43:55.975382+0000","last_undegraded":"2024-09-13T16:43:55.975382+0000","last_fullsized":"2024-09-13T16:43:55.975382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508820+0000","last_change":"2024-09-13T16:43:54.998229+0000","last_active":"2024-09-13T16:43:56.508820+0000","last_peered":"2024-09-13T16:43:56.508820+0000","last_clean":"2024-09-13T16:43:56.508820+0000","last_became_active":"2024-09-13T16:43:54.995774+0000","last_became_peered":"2024-09-13T16:43:54.995774+0000","last_unstale":"2024-09-13T16:43:56.508820+0000","last_undegraded":"2024-09-13T16:43:56.508820+0000","last_fullsized":"2024-09-13T16:43:56.508820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872235+0000","last_change":"2024-09-13T16:43:55.002126+0000","last_active":"2024-09-13T16:43:56.872235+0000","last_peered":"2024-09-13T16:43:56.872235+0000","last_clean":"2024-09-13T16:43:56.872235+0000","last_became_active":"2024-09-13T16:43:55.001713+0000","last_became_peered":"2024-09-13T16:43:55.001713+0000","last_unstale":"2024-09-13T16:43:56.872235+0000","last_undegraded":"2024-09-13T16:43:56.872235+0000","last_fullsized":"2024-09-13T16:43:56.872235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485939+0000","last_change":"2024-09-13T16:43:54.992745+0000","last_active":"2024-09-13T16:43:55.485939+0000","last_peered":"2024-09-13T16:43:55.485939+0000","last_clean":"2024-09-13T16:43:55.485939+0000","last_became_active":"2024-09-13T16:43:54.992508+0000","last_became_peered":"2024-09-13T16:43:54.992508+0000","last_unstale":"2024-09-13T16:43:55.485939+0000","last_undegraded":"2024-09-13T16:43:55.485939+0000","last_fullsized":"2024-09-13T16:43:55.485939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:23.913470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.979793+0000","last_change":"2024-09-13T16:43:54.990631+0000","last_active":"2024-09-13T16:43:55.979793+0000","last_peered":"2024-09-13T16:43:55.979793+0000","last_clean":"2024-09-13T16:43:55.979793+0000","last_became_active":"2024-09-13T16:43:54.986277+0000","last_became_peered":"2024-09-13T16:43:54.986277+0000","last_unstale":"2024-09-13T16:43:55.979793+0000","last_undegraded":"2024-09-13T16:43:55.979793+0000","last_fullsized":"2024-09-13T16:43:55.979793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260674+0000","last_change":"2024-09-13T16:43:55.000059+0000","last_active":"2024-09-13T16:43:56.260674+0000","last_peered":"2024-09-13T16:43:56.260674+0000","last_clean":"2024-09-13T16:43:56.260674+0000","last_became_active":"2024-09-13T16:43:54.999778+0000","last_became_peered":"2024-09-13T16:43:54.999778+0000","last_unstale":"2024-09-13T16:43:56.260674+0000","last_undegraded":"2024-09-13T16:43:56.260674+0000","last_fullsized":"2024-09-13T16:43:56.260674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935551+0000","last_change":"2024-09-13T16:43:54.988375+0000","last_active":"2024-09-13T16:43:56.935551+0000","last_peered":"2024-09-13T16:43:56.935551+0000","last_clean":"2024-09-13T16:43:56.935551+0000","last_became_active":"2024-09-13T16:43:54.988131+0000","last_became_peered":"2024-09-13T16:43:54.988131+0000","last_unstale":"2024-09-13T16:43:56.935551+0000","last_undegraded":"2024-09-13T16:43:56.935551+0000","last_fullsized":"2024-09-13T16:43:56.935551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.635755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:09.023856+0000","last_change":"2024-09-13T16:43:54.555042+0000","last_active":"2024-09-13T16:44:09.023856+0000","last_peered":"2024-09-13T16:44:09.023856+0000","last_clean":"2024-09-13T16:44:09.023856+0000","last_became_active":"2024-09-13T16:43:54.554697+0000","last_became_peered":"2024-09-13T16:43:54.554697+0000","last_unstale":"2024-09-13T16:44:09.023856+0000","last_undegraded":"2024-09-13T16:44:09.023856+0000","last_fullsized":"2024-09-13T16:44:09.023856+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872075+0000","last_change":"2024-09-13T16:43:54.998755+0000","last_active":"2024-09-13T16:43:56.872075+0000","last_peered":"2024-09-13T16:43:56.872075+0000","last_clean":"2024-09-13T16:43:56.872075+0000","last_became_active":"2024-09-13T16:43:54.998399+0000","last_became_peered":"2024-09-13T16:43:54.998399+0000","last_unstale":"2024-09-13T16:43:56.872075+0000","last_undegraded":"2024-09-13T16:43:56.872075+0000","last_fullsized":"2024-09-13T16:43:56.872075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.675242+0000","last_change":"2024-09-13T16:43:54.553250+0000","last_active":"2024-09-13T16:44:19.675242+0000","last_peered":"2024-09-13T16:44:19.675242+0000","last_clean":"2024-09-13T16:44:19.675242+0000","last_became_active":"2024-09-13T16:43:54.552977+0000","last_became_peered":"2024-09-13T16:43:54.552977+0000","last_unstale":"2024-09-13T16:44:19.675242+0000","last_undegraded":"2024-09-13T16:44:19.675242+0000","last_fullsized":"2024-09-13T16:44:19.675242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:14.094591+0000","last_change":"2024-09-13T16:43:54.558065+0000","last_active":"2024-09-13T16:44:14.094591+0000","last_peered":"2024-09-13T16:44:14.094591+0000","last_clean":"2024-09-13T16:44:14.094591+0000","last_became_active":"2024-09-13T16:43:54.557722+0000","last_became_peered":"2024-09-13T16:43:54.557722+0000","last_unstale":"2024-09-13T16:44:14.094591+0000","last_undegraded":"2024-09-13T16:44:14.094591+0000","last_fullsized":"2024-09-13T16:44:14.094591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508519+0000","last_change":"2024-09-13T16:43:55.002041+0000","last_active":"2024-09-13T16:43:56.508519+0000","last_peered":"2024-09-13T16:43:56.508519+0000","last_clean":"2024-09-13T16:43:56.508519+0000","last_became_active":"2024-09-13T16:43:55.001512+0000","last_became_peered":"2024-09-13T16:43:55.001512+0000","last_unstale":"2024-09-13T16:43:56.508519+0000","last_undegraded":"2024-09-13T16:43:56.508519+0000","last_fullsized":"2024-09-13T16:43:56.508519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.680255+0000","last_change":"2024-09-13T16:43:54.550389+0000","last_active":"2024-09-13T16:44:19.680255+0000","last_peered":"2024-09-13T16:44:19.680255+0000","last_clean":"2024-09-13T16:44:19.680255+0000","last_became_active":"2024-09-13T16:43:54.550154+0000","last_became_peered":"2024-09-13T16:43:54.550154+0000","last_unstale":"2024-09-13T16:44:19.680255+0000","last_undegraded":"2024-09-13T16:44:19.680255+0000","last_fullsized":"2024-09-13T16:44:19.680255+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975641+0000","last_change":"2024-09-13T16:43:54.993377+0000","last_active":"2024-09-13T16:43:55.975641+0000","last_peered":"2024-09-13T16:43:55.975641+0000","last_clean":"2024-09-13T16:43:55.975641+0000","last_became_active":"2024-09-13T16:43:54.993036+0000","last_became_peered":"2024-09-13T16:43:54.993036+0000","last_unstale":"2024-09-13T16:43:55.975641+0000","last_undegraded":"2024-09-13T16:43:55.975641+0000","last_fullsized":"2024-09-13T16:43:55.975641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:09.022198+0000","last_change":"2024-09-13T16:43:54.555528+0000","last_active":"2024-09-13T16:44:09.022198+0000","last_peered":"2024-09-13T16:44:09.022198+0000","last_clean":"2024-09-13T16:44:09.022198+0000","last_became_active":"2024-09-13T16:43:54.555267+0000","last_became_peered":"2024-09-13T16:43:54.555267+0000","last_unstale":"2024-09-13T16:44:09.022198+0000","last_undegraded":"2024-09-13T16:44:09.022198+0000","last_fullsized":"2024-09-13T16:44:09.022198+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486032+0000","last_change":"2024-09-13T16:43:54.981706+0000","last_active":"2024-09-13T16:43:55.486032+0000","last_peered":"2024-09-13T16:43:55.486032+0000","last_clean":"2024-09-13T16:43:55.486032+0000","last_became_active":"2024-09-13T16:43:54.981429+0000","last_became_peered":"2024-09-13T16:43:54.981429+0000","last_unstale":"2024-09-13T16:43:55.486032+0000","last_undegraded":"2024-09-13T16:43:55.486032+0000","last_fullsized":"2024-09-13T16:43:55.486032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.680087+0000","last_change":"2024-09-13T16:43:54.554593+0000","last_active":"2024-09-13T16:44:25.680087+0000","last_peered":"2024-09-13T16:44:25.680087+0000","last_clean":"2024-09-13T16:44:25.680087+0000","last_became_active":"2024-09-13T16:43:54.554307+0000","last_became_peered":"2024-09-13T16:43:54.554307+0000","last_unstale":"2024-09-13T16:44:25.680087+0000","last_undegraded":"2024-09-13T16:44:25.680087+0000","last_fullsized":"2024-09-13T16:44:25.680087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872139+0000","last_change":"2024-09-13T16:43:54.993485+0000","last_active":"2024-09-13T16:43:56.872139+0000","last_peered":"2024-09-13T16:43:56.872139+0000","last_clean":"2024-09-13T16:43:56.872139+0000","last_became_active":"2024-09-13T16:43:54.993226+0000","last_became_peered":"2024-09-13T16:43:54.993226+0000","last_unstale":"2024-09-13T16:43:56.872139+0000","last_undegraded":"2024-09-13T16:43:56.872139+0000","last_fullsized":"2024-09-13T16:43:56.872139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:09.023221+0000","last_change":"2024-09-13T16:43:54.562976+0000","last_active":"2024-09-13T16:44:09.023221+0000","last_peered":"2024-09-13T16:44:09.023221+0000","last_clean":"2024-09-13T16:44:09.023221+0000","last_became_active":"2024-09-13T16:43:54.560480+0000","last_became_peered":"2024-09-13T16:43:54.560480+0000","last_unstale":"2024-09-13T16:44:09.023221+0000","last_undegraded":"2024-09-13T16:44:09.023221+0000","last_fullsized":"2024-09-13T16:44:09.023221+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935689+0000","last_change":"2024-09-13T16:43:54.986459+0000","last_active":"2024-09-13T16:43:56.935689+0000","last_peered":"2024-09-13T16:43:56.935689+0000","last_clean":"2024-09-13T16:43:56.935689+0000","last_became_active":"2024-09-13T16:43:54.986083+0000","last_became_peered":"2024-09-13T16:43:54.986083+0000","last_unstale":"2024-09-13T16:43:56.935689+0000","last_undegraded":"2024-09-13T16:43:56.935689+0000","last_fullsized":"2024-09-13T16:43:56.935689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:51.960459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'10","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.678669+0000","last_change":"2024-09-13T16:43:54.565929+0000","last_active":"2024-09-13T16:44:25.678669+0000","last_peered":"2024-09-13T16:44:25.678669+0000","last_clean":"2024-09-13T16:44:25.678669+0000","last_became_active":"2024-09-13T16:43:54.565701+0000","last_became_peered":"2024-09-13T16:43:54.565701+0000","last_unstale":"2024-09-13T16:44:25.678669+0000","last_undegraded":"2024-09-13T16:44:25.678669+0000","last_fullsized":"2024-09-13T16:44:25.678669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178232+0000","last_change":"2024-09-13T16:43:54.989068+0000","last_active":"2024-09-13T16:43:59.178232+0000","last_peered":"2024-09-13T16:43:59.178232+0000","last_clean":"2024-09-13T16:43:59.178232+0000","last_became_active":"2024-09-13T16:43:54.988779+0000","last_became_peered":"2024-09-13T16:43:54.988779+0000","last_unstale":"2024-09-13T16:43:59.178232+0000","last_undegraded":"2024-09-13T16:43:59.178232+0000","last_fullsized":"2024-09-13T16:43:59.178232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.682164+0000","last_change":"2024-09-13T16:43:54.561487+0000","last_active":"2024-09-13T16:44:25.682164+0000","last_peered":"2024-09-13T16:44:25.682164+0000","last_clean":"2024-09-13T16:44:25.682164+0000","last_became_active":"2024-09-13T16:43:54.561077+0000","last_became_peered":"2024-09-13T16:43:54.561077+0000","last_unstale":"2024-09-13T16:44:25.682164+0000","last_undegraded":"2024-09-13T16:44:25.682164+0000","last_fullsized":"2024-09-13T16:44:25.682164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872141+0000","last_change":"2024-09-13T16:43:54.999517+0000","last_active":"2024-09-13T16:43:56.872141+0000","last_peered":"2024-09-13T16:43:56.872141+0000","last_clean":"2024-09-13T16:43:56.872141+0000","last_became_active":"2024-09-13T16:43:54.999283+0000","last_became_peered":"2024-09-13T16:43:54.999283+0000","last_unstale":"2024-09-13T16:43:56.872141+0000","last_undegraded":"2024-09-13T16:43:56.872141+0000","last_fullsized":"2024-09-13T16:43:56.872141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:20.207775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.674779+0000","last_change":"2024-09-13T16:43:54.542871+0000","last_active":"2024-09-13T16:44:19.674779+0000","last_peered":"2024-09-13T16:44:19.674779+0000","last_clean":"2024-09-13T16:44:19.674779+0000","last_became_active":"2024-09-13T16:43:54.542548+0000","last_became_peered":"2024-09-13T16:43:54.542548+0000","last_unstale":"2024-09-13T16:44:19.674779+0000","last_undegraded":"2024-09-13T16:44:19.674779+0000","last_fullsized":"2024-09-13T16:44:19.674779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178181+0000","last_change":"2024-09-13T16:43:54.989168+0000","last_active":"2024-09-13T16:43:59.178181+0000","last_peered":"2024-09-13T16:43:59.178181+0000","last_clean":"2024-09-13T16:43:59.178181+0000","last_became_active":"2024-09-13T16:43:54.988716+0000","last_became_peered":"2024-09-13T16:43:54.988716+0000","last_unstale":"2024-09-13T16:43:59.178181+0000","last_undegraded":"2024-09-13T16:43:59.178181+0000","last_fullsized":"2024-09-13T16:43:59.178181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.670560+0000","last_change":"2024-09-13T16:43:54.551551+0000","last_active":"2024-09-13T16:44:19.670560+0000","last_peered":"2024-09-13T16:44:19.670560+0000","last_clean":"2024-09-13T16:44:19.670560+0000","last_became_active":"2024-09-13T16:43:54.551293+0000","last_became_peered":"2024-09-13T16:43:54.551293+0000","last_unstale":"2024-09-13T16:44:19.670560+0000","last_undegraded":"2024-09-13T16:44:19.670560+0000","last_fullsized":"2024-09-13T16:44:19.670560+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260677+0000","last_change":"2024-09-13T16:43:54.994516+0000","last_active":"2024-09-13T16:43:56.260677+0000","last_peered":"2024-09-13T16:43:56.260677+0000","last_clean":"2024-09-13T16:43:56.260677+0000","last_became_active":"2024-09-13T16:43:54.994245+0000","last_became_peered":"2024-09-13T16:43:54.994245+0000","last_unstale":"2024-09-13T16:43:56.260677+0000","last_undegraded":"2024-09-13T16:43:56.260677+0000","last_fullsized":"2024-09-13T16:43:56.260677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":28,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.673680+0000","last_change":"2024-09-13T16:43:54.562232+0000","last_active":"2024-09-13T16:44:19.673680+0000","last_peered":"2024-09-13T16:44:19.673680+0000","last_clean":"2024-09-13T16:44:19.673680+0000","last_became_active":"2024-09-13T16:43:54.561939+0000","last_became_peered":"2024-09-13T16:43:54.561939+0000","last_unstale":"2024-09-13T16:44:19.673680+0000","last_undegraded":"2024-09-13T16:44:19.673680+0000","last_fullsized":"2024-09-13T16:44:19.673680+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178047+0000","last_change":"2024-09-13T16:43:54.988439+0000","last_active":"2024-09-13T16:43:59.178047+0000","last_peered":"2024-09-13T16:43:59.178047+0000","last_clean":"2024-09-13T16:43:59.178047+0000","last_became_active":"2024-09-13T16:43:54.988172+0000","last_became_peered":"2024-09-13T16:43:54.988172+0000","last_unstale":"2024-09-13T16:43:59.178047+0000","last_undegraded":"2024-09-13T16:43:59.178047+0000","last_fullsized":"2024-09-13T16:43:59.178047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.986952+0000","last_change":"2024-09-13T16:43:54.559372+0000","last_active":"2024-09-13T16:43:59.986952+0000","last_peered":"2024-09-13T16:43:59.986952+0000","last_clean":"2024-09-13T16:43:59.986952+0000","last_became_active":"2024-09-13T16:43:54.559111+0000","last_became_peered":"2024-09-13T16:43:54.559111+0000","last_unstale":"2024-09-13T16:43:59.986952+0000","last_undegraded":"2024-09-13T16:43:59.986952+0000","last_fullsized":"2024-09-13T16:43:59.986952+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:40.399883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178451+0000","last_change":"2024-09-13T16:43:54.988313+0000","last_active":"2024-09-13T16:43:59.178451+0000","last_peered":"2024-09-13T16:43:59.178451+0000","last_clean":"2024-09-13T16:43:59.178451+0000","last_became_active":"2024-09-13T16:43:54.988059+0000","last_became_peered":"2024-09-13T16:43:54.988059+0000","last_unstale":"2024-09-13T16:43:59.178451+0000","last_undegraded":"2024-09-13T16:43:59.178451+0000","last_fullsized":"2024-09-13T16:43:59.178451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.678704+0000","last_change":"2024-09-13T16:43:54.566410+0000","last_active":"2024-09-13T16:44:25.678704+0000","last_peered":"2024-09-13T16:44:25.678704+0000","last_clean":"2024-09-13T16:44:25.678704+0000","last_became_active":"2024-09-13T16:43:54.565861+0000","last_became_peered":"2024-09-13T16:43:54.565861+0000","last_unstale":"2024-09-13T16:44:25.678704+0000","last_undegraded":"2024-09-13T16:44:25.678704+0000","last_fullsized":"2024-09-13T16:44:25.678704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178547+0000","last_change":"2024-09-13T16:43:54.989506+0000","last_active":"2024-09-13T16:43:59.178547+0000","last_peered":"2024-09-13T16:43:59.178547+0000","last_clean":"2024-09-13T16:43:59.178547+0000","last_became_active":"2024-09-13T16:43:54.989233+0000","last_became_peered":"2024-09-13T16:43:54.989233+0000","last_unstale":"2024-09-13T16:43:59.178547+0000","last_undegraded":"2024-09-13T16:43:59.178547+0000","last_fullsized":"2024-09-13T16:43:59.178547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:23.913470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.675151+0000","last_change":"2024-09-13T16:43:54.559723+0000","last_active":"2024-09-13T16:44:19.675151+0000","last_peered":"2024-09-13T16:44:19.675151+0000","last_clean":"2024-09-13T16:44:19.675151+0000","last_became_active":"2024-09-13T16:43:54.559353+0000","last_became_peered":"2024-09-13T16:43:54.559353+0000","last_unstale":"2024-09-13T16:44:19.675151+0000","last_undegraded":"2024-09-13T16:44:19.675151+0000","last_fullsized":"2024-09-13T16:44:19.675151+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486000+0000","last_change":"2024-09-13T16:43:54.992511+0000","last_active":"2024-09-13T16:43:55.486000+0000","last_peered":"2024-09-13T16:43:55.486000+0000","last_clean":"2024-09-13T16:43:55.486000+0000","last_became_active":"2024-09-13T16:43:54.992249+0000","last_became_peered":"2024-09-13T16:43:54.992249+0000","last_unstale":"2024-09-13T16:43:55.486000+0000","last_undegraded":"2024-09-13T16:43:55.486000+0000","last_fullsized":"2024-09-13T16:43:55.486000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.675039+0000","last_change":"2024-09-13T16:43:54.551390+0000","last_active":"2024-09-13T16:44:19.675039+0000","last_peered":"2024-09-13T16:44:19.675039+0000","last_clean":"2024-09-13T16:44:19.675039+0000","last_became_active":"2024-09-13T16:43:54.547074+0000","last_became_peered":"2024-09-13T16:43:54.547074+0000","last_unstale":"2024-09-13T16:44:19.675039+0000","last_undegraded":"2024-09-13T16:44:19.675039+0000","last_fullsized":"2024-09-13T16:44:19.675039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178318+0000","last_change":"2024-09-13T16:43:54.990458+0000","last_active":"2024-09-13T16:43:59.178318+0000","last_peered":"2024-09-13T16:43:59.178318+0000","last_clean":"2024-09-13T16:43:59.178318+0000","last_became_active":"2024-09-13T16:43:54.990196+0000","last_became_peered":"2024-09-13T16:43:54.990196+0000","last_unstale":"2024-09-13T16:43:59.178318+0000","last_undegraded":"2024-09-13T16:43:59.178318+0000","last_fullsized":"2024-09-13T16:43:59.178318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981787+0000","last_change":"2024-09-13T16:43:51.914639+0000","last_active":"2024-09-13T16:43:54.981787+0000","last_peered":"2024-09-13T16:43:54.981787+0000","last_clean":"2024-09-13T16:43:54.981787+0000","last_became_active":"2024-09-13T16:43:51.913797+0000","last_became_peered":"2024-09-13T16:43:51.913797+0000","last_unstale":"2024-09-13T16:43:54.981787+0000","last_undegraded":"2024-09-13T16:43:54.981787+0000","last_fullsized":"2024-09-13T16:43:54.981787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_clean_scrub_stamp":"2024-09-13T16:43:50.498116+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:12.730911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.672667+0000","last_change":"2024-09-13T16:43:54.564153+0000","last_active":"2024-09-13T16:44:19.672667+0000","last_peered":"2024-09-13T16:44:19.672667+0000","last_clean":"2024-09-13T16:44:19.672667+0000","last_became_active":"2024-09-13T16:43:54.563949+0000","last_became_peered":"2024-09-13T16:43:54.563949+0000","last_unstale":"2024-09-13T16:44:19.672667+0000","last_undegraded":"2024-09-13T16:44:19.672667+0000","last_fullsized":"2024-09-13T16:44:19.672667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508836+0000","last_change":"2024-09-13T16:43:54.997576+0000","last_active":"2024-09-13T16:43:56.508836+0000","last_peered":"2024-09-13T16:43:56.508836+0000","last_clean":"2024-09-13T16:43:56.508836+0000","last_became_active":"2024-09-13T16:43:54.997315+0000","last_became_peered":"2024-09-13T16:43:54.997315+0000","last_unstale":"2024-09-13T16:43:56.508836+0000","last_undegraded":"2024-09-13T16:43:56.508836+0000","last_fullsized":"2024-09-13T16:43:56.508836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:17.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:14.094371+0000","last_change":"2024-09-13T16:43:54.564804+0000","last_active":"2024-09-13T16:44:14.094371+0000","last_peered":"2024-09-13T16:44:14.094371+0000","last_clean":"2024-09-13T16:44:14.094371+0000","last_became_active":"2024-09-13T16:43:54.564605+0000","last_became_peered":"2024-09-13T16:43:54.564605+0000","last_unstale":"2024-09-13T16:44:14.094371+0000","last_undegraded":"2024-09-13T16:44:14.094371+0000","last_fullsized":"2024-09-13T16:44:14.094371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.871801+0000","last_change":"2024-09-13T16:43:54.997884+0000","last_active":"2024-09-13T16:43:56.871801+0000","last_peered":"2024-09-13T16:43:56.871801+0000","last_clean":"2024-09-13T16:43:56.871801+0000","last_became_active":"2024-09-13T16:43:54.997510+0000","last_became_peered":"2024-09-13T16:43:54.997510+0000","last_unstale":"2024-09-13T16:43:56.871801+0000","last_undegraded":"2024-09-13T16:43:56.871801+0000","last_fullsized":"2024-09-13T16:43:56.871801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:14.094845+0000","last_change":"2024-09-13T16:43:54.566294+0000","last_active":"2024-09-13T16:44:14.094845+0000","last_peered":"2024-09-13T16:44:14.094845+0000","last_clean":"2024-09-13T16:44:14.094845+0000","last_became_active":"2024-09-13T16:43:54.566023+0000","last_became_peered":"2024-09-13T16:43:54.566023+0000","last_unstale":"2024-09-13T16:44:14.094845+0000","last_undegraded":"2024-09-13T16:44:14.094845+0000","last_fullsized":"2024-09-13T16:44:14.094845+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975316+0000","last_change":"2024-09-13T16:43:54.981116+0000","last_active":"2024-09-13T16:43:55.975316+0000","last_peered":"2024-09-13T16:43:55.975316+0000","last_clean":"2024-09-13T16:43:55.975316+0000","last_became_active":"2024-09-13T16:43:54.980766+0000","last_became_peered":"2024-09-13T16:43:54.980766+0000","last_unstale":"2024-09-13T16:43:55.975316+0000","last_undegraded":"2024-09-13T16:43:55.975316+0000","last_fullsized":"2024-09-13T16:43:55.975316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.682397+0000","last_change":"2024-09-13T16:43:54.563716+0000","last_active":"2024-09-13T16:44:25.682397+0000","last_peered":"2024-09-13T16:44:25.682397+0000","last_clean":"2024-09-13T16:44:25.682397+0000","last_became_active":"2024-09-13T16:43:54.563522+0000","last_became_peered":"2024-09-13T16:43:54.563522+0000","last_unstale":"2024-09-13T16:44:25.682397+0000","last_undegraded":"2024-09-13T16:44:25.682397+0000","last_fullsized":"2024-09-13T16:44:25.682397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935617+0000","last_change":"2024-09-13T16:43:54.999051+0000","last_active":"2024-09-13T16:43:56.935617+0000","last_peered":"2024-09-13T16:43:56.935617+0000","last_clean":"2024-09-13T16:43:56.935617+0000","last_became_active":"2024-09-13T16:43:54.998803+0000","last_became_peered":"2024-09-13T16:43:54.998803+0000","last_unstale":"2024-09-13T16:43:56.935617+0000","last_undegraded":"2024-09-13T16:43:56.935617+0000","last_fullsized":"2024-09-13T16:43:56.935617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975558+0000","last_change":"2024-09-13T16:43:54.569945+0000","last_active":"2024-09-13T16:43:55.975558+0000","last_peered":"2024-09-13T16:43:55.975558+0000","last_clean":"2024-09-13T16:43:55.975558+0000","last_became_active":"2024-09-13T16:43:54.568846+0000","last_became_peered":"2024-09-13T16:43:54.568846+0000","last_unstale":"2024-09-13T16:43:55.975558+0000","last_undegraded":"2024-09-13T16:43:55.975558+0000","last_fullsized":"2024-09-13T16:43:55.975558+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486021+0000","last_change":"2024-09-13T16:43:54.992082+0000","last_active":"2024-09-13T16:43:55.486021+0000","last_peered":"2024-09-13T16:43:55.486021+0000","last_clean":"2024-09-13T16:43:55.486021+0000","last_became_active":"2024-09-13T16:43:54.991659+0000","last_became_peered":"2024-09-13T16:43:54.991659+0000","last_unstale":"2024-09-13T16:43:55.486021+0000","last_undegraded":"2024-09-13T16:43:55.486021+0000","last_fullsized":"2024-09-13T16:43:55.486021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.676205+0000","last_change":"2024-09-13T16:43:54.569111+0000","last_active":"2024-09-13T16:44:19.676205+0000","last_peered":"2024-09-13T16:44:19.676205+0000","last_clean":"2024-09-13T16:44:19.676205+0000","last_became_active":"2024-09-13T16:43:54.568903+0000","last_became_peered":"2024-09-13T16:43:54.568903+0000","last_unstale":"2024-09-13T16:44:19.676205+0000","last_undegraded":"2024-09-13T16:44:19.676205+0000","last_fullsized":"2024-09-13T16:44:19.676205+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975475+0000","last_change":"2024-09-13T16:43:54.980100+0000","last_active":"2024-09-13T16:43:55.975475+0000","last_peered":"2024-09-13T16:43:55.975475+0000","last_clean":"2024-09-13T16:43:55.975475+0000","last_became_active":"2024-09-13T16:43:54.979584+0000","last_became_peered":"2024-09-13T16:43:54.979584+0000","last_unstale":"2024-09-13T16:43:55.975475+0000","last_undegraded":"2024-09-13T16:43:55.975475+0000","last_fullsized":"2024-09-13T16:43:55.975475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'7","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.673990+0000","last_change":"2024-09-13T16:43:54.560710+0000","last_active":"2024-09-13T16:44:19.673990+0000","last_peered":"2024-09-13T16:44:19.673990+0000","last_clean":"2024-09-13T16:44:19.673990+0000","last_became_active":"2024-09-13T16:43:54.560030+0000","last_became_peered":"2024-09-13T16:43:54.560030+0000","last_unstale":"2024-09-13T16:44:19.673990+0000","last_undegraded":"2024-09-13T16:44:19.673990+0000","last_fullsized":"2024-09-13T16:44:19.673990+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.509048+0000","last_change":"2024-09-13T16:43:54.990812+0000","last_active":"2024-09-13T16:43:56.509048+0000","last_peered":"2024-09-13T16:43:56.509048+0000","last_clean":"2024-09-13T16:43:56.509048+0000","last_became_active":"2024-09-13T16:43:54.990440+0000","last_became_peered":"2024-09-13T16:43:54.990440+0000","last_unstale":"2024-09-13T16:43:56.509048+0000","last_undegraded":"2024-09-13T16:43:56.509048+0000","last_fullsized":"2024-09-13T16:43:56.509048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.682745+0000","last_change":"2024-09-13T16:43:54.558439+0000","last_active":"2024-09-13T16:44:25.682745+0000","last_peered":"2024-09-13T16:44:25.682745+0000","last_clean":"2024-09-13T16:44:25.682745+0000","last_became_active":"2024-09-13T16:43:54.558194+0000","last_became_peered":"2024-09-13T16:43:54.558194+0000","last_unstale":"2024-09-13T16:44:25.682745+0000","last_undegraded":"2024-09-13T16:44:25.682745+0000","last_fullsized":"2024-09-13T16:44:25.682745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872087+0000","last_change":"2024-09-13T16:43:54.995534+0000","last_active":"2024-09-13T16:43:56.872087+0000","last_peered":"2024-09-13T16:43:56.872087+0000","last_clean":"2024-09-13T16:43:56.872087+0000","last_became_active":"2024-09-13T16:43:54.995397+0000","last_became_peered":"2024-09-13T16:43:54.995397+0000","last_unstale":"2024-09-13T16:43:56.872087+0000","last_undegraded":"2024-09-13T16:43:56.872087+0000","last_fullsized":"2024-09-13T16:43:56.872087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:19.697247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:09.021581+0000","last_change":"2024-09-13T16:43:54.569085+0000","last_active":"2024-09-13T16:44:09.021581+0000","last_peered":"2024-09-13T16:44:09.021581+0000","last_clean":"2024-09-13T16:44:09.021581+0000","last_became_active":"2024-09-13T16:43:54.568892+0000","last_became_peered":"2024-09-13T16:43:54.568892+0000","last_unstale":"2024-09-13T16:44:09.021581+0000","last_undegraded":"2024-09-13T16:44:09.021581+0000","last_fullsized":"2024-09-13T16:44:09.021581+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935459+0000","last_change":"2024-09-13T16:43:54.984120+0000","last_active":"2024-09-13T16:43:56.935459+0000","last_peered":"2024-09-13T16:43:56.935459+0000","last_clean":"2024-09-13T16:43:56.935459+0000","last_became_active":"2024-09-13T16:43:54.983886+0000","last_became_peered":"2024-09-13T16:43:54.983886+0000","last_unstale":"2024-09-13T16:43:56.935459+0000","last_undegraded":"2024-09-13T16:43:56.935459+0000","last_fullsized":"2024-09-13T16:43:56.935459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.679742+0000","last_change":"2024-09-13T16:43:54.555092+0000","last_active":"2024-09-13T16:44:25.679742+0000","last_peered":"2024-09-13T16:44:25.679742+0000","last_clean":"2024-09-13T16:44:25.679742+0000","last_became_active":"2024-09-13T16:43:54.554230+0000","last_became_peered":"2024-09-13T16:43:54.554230+0000","last_unstale":"2024-09-13T16:44:25.679742+0000","last_undegraded":"2024-09-13T16:44:25.679742+0000","last_fullsized":"2024-09-13T16:44:25.679742+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508681+0000","last_change":"2024-09-13T16:43:54.984588+0000","last_active":"2024-09-13T16:43:56.508681+0000","last_peered":"2024-09-13T16:43:56.508681+0000","last_clean":"2024-09-13T16:43:56.508681+0000","last_became_active":"2024-09-13T16:43:54.984221+0000","last_became_peered":"2024-09-13T16:43:54.984221+0000","last_unstale":"2024-09-13T16:43:56.508681+0000","last_undegraded":"2024-09-13T16:43:56.508681+0000","last_fullsized":"2024-09-13T16:43:56.508681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.673474+0000","last_change":"2024-09-13T16:43:54.562646+0000","last_active":"2024-09-13T16:44:19.673474+0000","last_peered":"2024-09-13T16:44:19.673474+0000","last_clean":"2024-09-13T16:44:19.673474+0000","last_became_active":"2024-09-13T16:43:54.556888+0000","last_became_peered":"2024-09-13T16:43:54.556888+0000","last_unstale":"2024-09-13T16:44:19.673474+0000","last_undegraded":"2024-09-13T16:44:19.673474+0000","last_fullsized":"2024-09-13T16:44:19.673474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508792+0000","last_change":"2024-09-13T16:43:54.999328+0000","last_active":"2024-09-13T16:43:56.508792+0000","last_peered":"2024-09-13T16:43:56.508792+0000","last_clean":"2024-09-13T16:43:56.508792+0000","last_became_active":"2024-09-13T16:43:54.999101+0000","last_became_peered":"2024-09-13T16:43:54.999101+0000","last_unstale":"2024-09-13T16:43:56.508792+0000","last_undegraded":"2024-09-13T16:43:56.508792+0000","last_fullsized":"2024-09-13T16:43:56.508792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:09.021554+0000","last_change":"2024-09-13T16:43:54.559214+0000","last_active":"2024-09-13T16:44:09.021554+0000","last_peered":"2024-09-13T16:44:09.021554+0000","last_clean":"2024-09-13T16:44:09.021554+0000","last_became_active":"2024-09-13T16:43:54.558233+0000","last_became_peered":"2024-09-13T16:43:54.558233+0000","last_unstale":"2024-09-13T16:44:09.021554+0000","last_undegraded":"2024-09-13T16:44:09.021554+0000","last_fullsized":"2024-09-13T16:44:09.021554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260789+0000","last_change":"2024-09-13T16:43:54.986565+0000","last_active":"2024-09-13T16:43:56.260789+0000","last_peered":"2024-09-13T16:43:56.260789+0000","last_clean":"2024-09-13T16:43:56.260789+0000","last_became_active":"2024-09-13T16:43:54.986084+0000","last_became_peered":"2024-09-13T16:43:54.986084+0000","last_unstale":"2024-09-13T16:43:56.260789+0000","last_undegraded":"2024-09-13T16:43:56.260789+0000","last_fullsized":"2024-09-13T16:43:56.260789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486010+0000","last_change":"2024-09-13T16:43:54.982611+0000","last_active":"2024-09-13T16:43:55.486010+0000","last_peered":"2024-09-13T16:43:55.486010+0000","last_clean":"2024-09-13T16:43:55.486010+0000","last_became_active":"2024-09-13T16:43:54.982338+0000","last_became_peered":"2024-09-13T16:43:54.982338+0000","last_unstale":"2024-09-13T16:43:55.486010+0000","last_undegraded":"2024-09-13T16:43:55.486010+0000","last_fullsized":"2024-09-13T16:43:55.486010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.682678+0000","last_change":"2024-09-13T16:43:54.558799+0000","last_active":"2024-09-13T16:44:25.682678+0000","last_peered":"2024-09-13T16:44:25.682678+0000","last_clean":"2024-09-13T16:44:25.682678+0000","last_became_active":"2024-09-13T16:43:54.558332+0000","last_became_peered":"2024-09-13T16:43:54.558332+0000","last_unstale":"2024-09-13T16:44:25.682678+0000","last_undegraded":"2024-09-13T16:44:25.682678+0000","last_fullsized":"2024-09-13T16:44:25.682678+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:19.201506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485991+0000","last_change":"2024-09-13T16:43:54.984150+0000","last_active":"2024-09-13T16:43:55.485991+0000","last_peered":"2024-09-13T16:43:55.485991+0000","last_clean":"2024-09-13T16:43:55.485991+0000","last_became_active":"2024-09-13T16:43:54.983881+0000","last_became_peered":"2024-09-13T16:43:54.983881+0000","last_unstale":"2024-09-13T16:43:55.485991+0000","last_undegraded":"2024-09-13T16:43:55.485991+0000","last_fullsized":"2024-09-13T16:43:55.485991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:29.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'12","reported_seq":23,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.682774+0000","last_change":"2024-09-13T16:43:54.559551+0000","last_active":"2024-09-13T16:44:25.682774+0000","last_peered":"2024-09-13T16:44:25.682774+0000","last_clean":"2024-09-13T16:44:25.682774+0000","last_became_active":"2024-09-13T16:43:54.559290+0000","last_became_peered":"2024-09-13T16:43:54.559290+0000","last_unstale":"2024-09-13T16:44:25.682774+0000","last_undegraded":"2024-09-13T16:44:25.682774+0000","last_fullsized":"2024-09-13T16:44:25.682774+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872084+0000","last_change":"2024-09-13T16:43:55.000783+0000","last_active":"2024-09-13T16:43:56.872084+0000","last_peered":"2024-09-13T16:43:56.872084+0000","last_clean":"2024-09-13T16:43:56.872084+0000","last_became_active":"2024-09-13T16:43:55.000196+0000","last_became_peered":"2024-09-13T16:43:55.000196+0000","last_unstale":"2024-09-13T16:43:56.872084+0000","last_undegraded":"2024-09-13T16:43:56.872084+0000","last_fullsized":"2024-09-13T16:43:56.872084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.986199+0000","last_change":"2024-09-13T16:43:54.559886+0000","last_active":"2024-09-13T16:43:59.986199+0000","last_peered":"2024-09-13T16:43:59.986199+0000","last_clean":"2024-09-13T16:43:59.986199+0000","last_became_active":"2024-09-13T16:43:54.559586+0000","last_became_peered":"2024-09-13T16:43:54.559586+0000","last_unstale":"2024-09-13T16:43:59.986199+0000","last_undegraded":"2024-09-13T16:43:59.986199+0000","last_fullsized":"2024-09-13T16:43:59.986199+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508978+0000","last_change":"2024-09-13T16:43:54.996099+0000","last_active":"2024-09-13T16:43:56.508978+0000","last_peered":"2024-09-13T16:43:56.508978+0000","last_clean":"2024-09-13T16:43:56.508978+0000","last_became_active":"2024-09-13T16:43:54.995802+0000","last_became_peered":"2024-09-13T16:43:54.995802+0000","last_unstale":"2024-09-13T16:43:56.508978+0000","last_undegraded":"2024-09-13T16:43:56.508978+0000","last_fullsized":"2024-09-13T16:43:56.508978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":24,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.679752+0000","last_change":"2024-09-13T16:43:54.562699+0000","last_active":"2024-09-13T16:44:25.679752+0000","last_peered":"2024-09-13T16:44:25.679752+0000","last_clean":"2024-09-13T16:44:25.679752+0000","last_became_active":"2024-09-13T16:43:54.560552+0000","last_became_peered":"2024-09-13T16:43:54.560552+0000","last_unstale":"2024-09-13T16:44:25.679752+0000","last_undegraded":"2024-09-13T16:44:25.679752+0000","last_fullsized":"2024-09-13T16:44:25.679752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:11.032382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7794,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":15588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":307200,"data_stored":663318,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":73204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":74120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":72180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":73238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":71698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":72382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":294912,"data_stored":661121,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:44:38.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:44:38.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.565+0000 7f265b000640 1 Processor -- start 2024-09-13T16:44:38.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.565+0000 7f265b000640 1 -- start start 2024-09-13T16:44:38.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.567+0000 7f265b000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f26540c3670 0x7f26540c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:38.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.567+0000 7f265b000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f265414ca00 con 0x7f26540c5bd0 2024-09-13T16:44:38.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.567+0000 7f265b000640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2654155990 con 0x7f26540c1ea0 2024-09-13T16:44:38.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.567+0000 7f265b000640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26540c2dc0 con 0x7f26540c3670 2024-09-13T16:44:38.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.567+0000 7f2653e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f26540c5bd0 0x7f2654156150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53806/0 (socket says 172.21.15.28:53806) 2024-09-13T16:44:38.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.567+0000 7f2653e00640 1 -- 172.21.15.28:0/2107308598 learned_addr learned my addr 172.21.15.28:0/2107308598 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:38.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.567+0000 7f264be00640 1 --2- 172.21.15.28:0/2107308598 >> v2:172.21.15.191:3300/0 conn(0x7f26540c3670 0x7f26540c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:38.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.567+0000 7f2653400640 1 -- 172.21.15.28:0/2107308598 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383000244 0 0) 0x7f265414ca00 con 0x7f26540c5bd0 2024-09-13T16:44:38.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.567+0000 7f2653400640 1 -- 172.21.15.28:0/2107308598 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2638003520 con 0x7f26540c5bd0 2024-09-13T16:44:38.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.568+0000 7f2653400640 1 -- 172.21.15.28:0/2107308598 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740319686 0 0) 0x7f2654155990 con 0x7f26540c1ea0 2024-09-13T16:44:38.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.568+0000 7f264be00640 1 -- 172.21.15.28:0/2107308598 >> v1:172.21.15.191:6789/0 conn(0x7f26540c1ea0 legacy=0x7f26540c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.568+0000 7f264be00640 1 -- 172.21.15.28:0/2107308598 >> v1:172.21.15.28:6789/0 conn(0x7f26540c5bd0 legacy=0x7f2654156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.568+0000 7f264be00640 1 -- 172.21.15.28:0/2107308598 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26540c3b60 con 0x7f26540c3670 2024-09-13T16:44:38.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.569+0000 7f264be00640 1 --2- 172.21.15.28:0/2107308598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26540c3670 0x7f26540c1960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f263c0097a0 tx=0x7f263c02a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1429c81db57b8902 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:38.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.569+0000 7f2653400640 1 -- 172.21.15.28:0/2107308598 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c032640 con 0x7f26540c3670 2024-09-13T16:44:38.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.569+0000 7f2653400640 1 -- 172.21.15.28:0/2107308598 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263c0327a0 con 0x7f26540c3670 2024-09-13T16:44:38.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.569+0000 7f2653400640 1 -- 172.21.15.28:0/2107308598 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c032b30 con 0x7f26540c3670 2024-09-13T16:44:38.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.569+0000 7f265b000640 1 -- 172.21.15.28:0/2107308598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26540c3670 msgr2=0x7f26540c1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.569+0000 7f265b000640 1 --2- 172.21.15.28:0/2107308598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26540c3670 0x7f26540c1960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f263c0097a0 tx=0x7f263c02a2c0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.569+0000 7f265b000640 1 -- 172.21.15.28:0/2107308598 shutdown_connections 2024-09-13T16:44:38.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.569+0000 7f265b000640 1 --2- 172.21.15.28:0/2107308598 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26540c3670 0x7f26540c1960 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.569+0000 7f265b000640 1 -- 172.21.15.28:0/2107308598 >> 172.21.15.28:0/2107308598 conn(0x7f2654066040 msgr2=0x7f2654069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:38.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.570+0000 7f265b000640 1 -- 172.21.15.28:0/2107308598 shutdown_connections 2024-09-13T16:44:38.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.570+0000 7f265b000640 1 -- 172.21.15.28:0/2107308598 wait complete. 2024-09-13T16:44:38.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.571+0000 7f265b000640 1 Processor -- start 2024-09-13T16:44:38.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.571+0000 7f265b000640 1 -- start start 2024-09-13T16:44:38.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.572+0000 7f265b000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26540c1ea0 0x7f26541debb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:38.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.572+0000 7f265b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540c3670 0x7f2654152b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:38.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.572+0000 7f265b000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26540c5bd0 0x7f2654144c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:38.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.572+0000 7f265b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2654150fa0 con 0x7f26540c3670 2024-09-13T16:44:38.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.572+0000 7f265b000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2654148090 con 0x7f26540c5bd0 2024-09-13T16:44:38.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.572+0000 7f265b000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2654146420 con 0x7f26540c1ea0 2024-09-13T16:44:38.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.572+0000 7f264be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540c3670 0x7f2654152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:38.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.572+0000 7f264be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540c3670 0x7f2654152b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44334/0 (socket says 172.21.15.28:44334) 2024-09-13T16:44:38.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.572+0000 7f264be00640 1 -- 172.21.15.28:0/1861689007 learned_addr learned my addr 172.21.15.28:0/1861689007 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:38.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.573+0000 7f2659000640 1 --2- 172.21.15.28:0/1861689007 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26540c5bd0 0x7f2654144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:38.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.573+0000 7f2653e00640 1 --2- 172.21.15.28:0/1861689007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26540c1ea0 0x7f26541debb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:38.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.573+0000 7f264be00640 1 -- 172.21.15.28:0/1861689007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26540c1ea0 msgr2=0x7f26541debb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.573+0000 7f264be00640 1 --2- 172.21.15.28:0/1861689007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26540c1ea0 0x7f26541debb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.573+0000 7f264be00640 1 -- 172.21.15.28:0/1861689007 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26540c5bd0 msgr2=0x7f2654144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.573+0000 7f264be00640 1 --2- 172.21.15.28:0/1861689007 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26540c5bd0 0x7f2654144c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.573+0000 7f264be00640 1 -- 172.21.15.28:0/1861689007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c009400 con 0x7f26540c3670 2024-09-13T16:44:38.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.573+0000 7f264be00640 1 --2- 172.21.15.28:0/1861689007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540c3670 0x7f2654152b50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f263c035040 tx=0x7f263c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:38.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.573+0000 7f2650c00640 1 -- 172.21.15.28:0/1861689007 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c03d8f0 con 0x7f26540c3670 2024-09-13T16:44:38.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.574+0000 7f2650c00640 1 -- 172.21.15.28:0/1861689007 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263c03da50 con 0x7f26540c3670 2024-09-13T16:44:38.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.574+0000 7f2650c00640 1 -- 172.21.15.28:0/1861689007 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c03dde0 con 0x7f26540c3670 2024-09-13T16:44:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.574+0000 7f265b000640 1 -- 172.21.15.28:0/1861689007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2654148350 con 0x7f26540c3670 2024-09-13T16:44:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.574+0000 7f265b000640 1 -- 172.21.15.28:0/1861689007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26541485c0 con 0x7f26540c3670 2024-09-13T16:44:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.575+0000 7f265b000640 1 -- 172.21.15.28:0/1861689007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26540c89c0 con 0x7f26540c3670 2024-09-13T16:44:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.576+0000 7f2650c00640 1 -- 172.21.15.28:0/1861689007 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f263c039040 con 0x7f26540c3670 2024-09-13T16:44:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.576+0000 7f2650c00640 1 --2- 172.21.15.28:0/1861689007 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f263006ac00 0x7f263006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.577+0000 7f2650c00640 1 -- 172.21.15.28:0/1861689007 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f263c0a9e70 con 0x7f26540c3670 2024-09-13T16:44:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.577+0000 7f2653e00640 1 --2- 172.21.15.28:0/1861689007 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f263006ac00 0x7f263006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.577+0000 7f2653e00640 1 --2- 172.21.15.28:0/1861689007 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f263006ac00 0x7f263006d0c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2644005880 tx=0x7f2644009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:38.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.581+0000 7f2650c00640 1 -- 172.21.15.28:0/1861689007 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f263c074fd0 con 0x7f26540c3670 2024-09-13T16:44:38.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.757+0000 7f265b000640 1 -- 172.21.15.28:0/1861689007 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f26540c4120 con 0x7f263006ac00 2024-09-13T16:44:38.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.764+0000 7f2650c00640 1 -- 172.21.15.28:0/1861689007 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264747 (secure 0 0 0) 0x7f264400f820 con 0x7f263006ac00 2024-09-13T16:44:38.769 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:38.770 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T16:44:38.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.768+0000 7f265b000640 1 -- 172.21.15.28:0/1861689007 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f263006ac00 msgr2=0x7f263006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.768+0000 7f265b000640 1 --2- 172.21.15.28:0/1861689007 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f263006ac00 0x7f263006d0c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2644005880 tx=0x7f2644009040 comp rx=0 tx=0).stop 2024-09-13T16:44:38.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.768+0000 7f265b000640 1 -- 172.21.15.28:0/1861689007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540c3670 msgr2=0x7f2654152b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.768+0000 7f265b000640 1 --2- 172.21.15.28:0/1861689007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540c3670 0x7f2654152b50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f263c035040 tx=0x7f263c0046c0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.768+0000 7f265b000640 1 -- 172.21.15.28:0/1861689007 shutdown_connections 2024-09-13T16:44:38.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.768+0000 7f265b000640 1 --2- 172.21.15.28:0/1861689007 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f263006ac00 0x7f263006d0c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.768+0000 7f265b000640 1 --2- 172.21.15.28:0/1861689007 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26540c5bd0 0x7f2654144c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.768+0000 7f265b000640 1 --2- 172.21.15.28:0/1861689007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540c3670 0x7f2654152b50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.768+0000 7f265b000640 1 --2- 172.21.15.28:0/1861689007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26540c1ea0 0x7f26541debb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.768+0000 7f265b000640 1 -- 172.21.15.28:0/1861689007 >> 172.21.15.28:0/1861689007 conn(0x7f2654066040 msgr2=0x7f26540c6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:38.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.768+0000 7f265b000640 1 -- 172.21.15.28:0/1861689007 shutdown_connections 2024-09-13T16:44:38.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.768+0000 7f265b000640 1 -- 172.21.15.28:0/1861689007 wait complete. 2024-09-13T16:44:38.785 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-09-13T16:44:37.601103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598162,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":364,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222196,"kb_used_data":1972,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754752524,"statfs":{"total":773094113280,"available":772866584576,"internally_reserved":0,"allocated":2019328,"data_stored":1761261,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12743,"internal_metadata":225431097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1358,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002968"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178474+0000","last_change":"2024-09-13T16:43:54.987241+0000","last_active":"2024-09-13T16:43:59.178474+0000","last_peered":"2024-09-13T16:43:59.178474+0000","last_clean":"2024-09-13T16:43:59.178474+0000","last_became_active":"2024-09-13T16:43:54.986988+0000","last_became_peered":"2024-09-13T16:43:54.986988+0000","last_unstale":"2024-09-13T16:43:59.178474+0000","last_undegraded":"2024-09-13T16:43:59.178474+0000","last_fullsized":"2024-09-13T16:43:59.178474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975685+0000","last_change":"2024-09-13T16:43:55.001141+0000","last_active":"2024-09-13T16:43:55.975685+0000","last_peered":"2024-09-13T16:43:55.975685+0000","last_clean":"2024-09-13T16:43:55.975685+0000","last_became_active":"2024-09-13T16:43:55.000757+0000","last_became_peered":"2024-09-13T16:43:55.000757+0000","last_unstale":"2024-09-13T16:43:55.975685+0000","last_undegraded":"2024-09-13T16:43:55.975685+0000","last_fullsized":"2024-09-13T16:43:55.975685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872108+0000","last_change":"2024-09-13T16:43:54.986925+0000","last_active":"2024-09-13T16:43:56.872108+0000","last_peered":"2024-09-13T16:43:56.872108+0000","last_clean":"2024-09-13T16:43:56.872108+0000","last_became_active":"2024-09-13T16:43:54.986198+0000","last_became_peered":"2024-09-13T16:43:54.986198+0000","last_unstale":"2024-09-13T16:43:56.872108+0000","last_undegraded":"2024-09-13T16:43:56.872108+0000","last_fullsized":"2024-09-13T16:43:56.872108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:03.127550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485981+0000","last_change":"2024-09-13T16:43:54.992249+0000","last_active":"2024-09-13T16:43:55.485981+0000","last_peered":"2024-09-13T16:43:55.485981+0000","last_clean":"2024-09-13T16:43:55.485981+0000","last_became_active":"2024-09-13T16:43:54.991804+0000","last_became_peered":"2024-09-13T16:43:54.991804+0000","last_unstale":"2024-09-13T16:43:55.485981+0000","last_undegraded":"2024-09-13T16:43:55.485981+0000","last_fullsized":"2024-09-13T16:43:55.485981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178163+0000","last_change":"2024-09-13T16:43:54.987564+0000","last_active":"2024-09-13T16:43:59.178163+0000","last_peered":"2024-09-13T16:43:59.178163+0000","last_clean":"2024-09-13T16:43:59.178163+0000","last_became_active":"2024-09-13T16:43:54.987328+0000","last_became_peered":"2024-09-13T16:43:54.987328+0000","last_unstale":"2024-09-13T16:43:59.178163+0000","last_undegraded":"2024-09-13T16:43:59.178163+0000","last_fullsized":"2024-09-13T16:43:59.178163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:36.006730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485919+0000","last_change":"2024-09-13T16:43:55.000019+0000","last_active":"2024-09-13T16:43:55.485919+0000","last_peered":"2024-09-13T16:43:55.485919+0000","last_clean":"2024-09-13T16:43:55.485919+0000","last_became_active":"2024-09-13T16:43:54.999752+0000","last_became_peered":"2024-09-13T16:43:54.999752+0000","last_unstale":"2024-09-13T16:43:55.485919+0000","last_undegraded":"2024-09-13T16:43:55.485919+0000","last_fullsized":"2024-09-13T16:43:55.485919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.177985+0000","last_change":"2024-09-13T16:43:54.999533+0000","last_active":"2024-09-13T16:43:59.177985+0000","last_peered":"2024-09-13T16:43:59.177985+0000","last_clean":"2024-09-13T16:43:59.177985+0000","last_became_active":"2024-09-13T16:43:54.999350+0000","last_became_peered":"2024-09-13T16:43:54.999350+0000","last_unstale":"2024-09-13T16:43:59.177985+0000","last_undegraded":"2024-09-13T16:43:59.177985+0000","last_fullsized":"2024-09-13T16:43:59.177985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:29.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508531+0000","last_change":"2024-09-13T16:43:54.998526+0000","last_active":"2024-09-13T16:43:56.508531+0000","last_peered":"2024-09-13T16:43:56.508531+0000","last_clean":"2024-09-13T16:43:56.508531+0000","last_became_active":"2024-09-13T16:43:54.996309+0000","last_became_peered":"2024-09-13T16:43:54.996309+0000","last_unstale":"2024-09-13T16:43:56.508531+0000","last_undegraded":"2024-09-13T16:43:56.508531+0000","last_fullsized":"2024-09-13T16:43:56.508531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178473+0000","last_change":"2024-09-13T16:43:54.993786+0000","last_active":"2024-09-13T16:43:59.178473+0000","last_peered":"2024-09-13T16:43:59.178473+0000","last_clean":"2024-09-13T16:43:59.178473+0000","last_became_active":"2024-09-13T16:43:54.993475+0000","last_became_peered":"2024-09-13T16:43:54.993475+0000","last_unstale":"2024-09-13T16:43:59.178473+0000","last_undegraded":"2024-09-13T16:43:59.178473+0000","last_fullsized":"2024-09-13T16:43:59.178473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260988+0000","last_change":"2024-09-13T16:43:54.986272+0000","last_active":"2024-09-13T16:43:56.260988+0000","last_peered":"2024-09-13T16:43:56.260988+0000","last_clean":"2024-09-13T16:43:56.260988+0000","last_became_active":"2024-09-13T16:43:54.985979+0000","last_became_peered":"2024-09-13T16:43:54.985979+0000","last_unstale":"2024-09-13T16:43:56.260988+0000","last_undegraded":"2024-09-13T16:43:56.260988+0000","last_fullsized":"2024-09-13T16:43:56.260988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260766+0000","last_change":"2024-09-13T16:43:54.986207+0000","last_active":"2024-09-13T16:43:56.260766+0000","last_peered":"2024-09-13T16:43:56.260766+0000","last_clean":"2024-09-13T16:43:56.260766+0000","last_became_active":"2024-09-13T16:43:54.985948+0000","last_became_peered":"2024-09-13T16:43:54.985948+0000","last_unstale":"2024-09-13T16:43:56.260766+0000","last_undegraded":"2024-09-13T16:43:56.260766+0000","last_fullsized":"2024-09-13T16:43:56.260766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935591+0000","last_change":"2024-09-13T16:43:54.986636+0000","last_active":"2024-09-13T16:43:56.935591+0000","last_peered":"2024-09-13T16:43:56.935591+0000","last_clean":"2024-09-13T16:43:56.935591+0000","last_became_active":"2024-09-13T16:43:54.986161+0000","last_became_peered":"2024-09-13T16:43:54.986161+0000","last_unstale":"2024-09-13T16:43:56.935591+0000","last_undegraded":"2024-09-13T16:43:56.935591+0000","last_fullsized":"2024-09-13T16:43:56.935591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:09.340330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508720+0000","last_change":"2024-09-13T16:43:54.997783+0000","last_active":"2024-09-13T16:43:56.508720+0000","last_peered":"2024-09-13T16:43:56.508720+0000","last_clean":"2024-09-13T16:43:56.508720+0000","last_became_active":"2024-09-13T16:43:54.997162+0000","last_became_peered":"2024-09-13T16:43:54.997162+0000","last_unstale":"2024-09-13T16:43:56.508720+0000","last_undegraded":"2024-09-13T16:43:56.508720+0000","last_fullsized":"2024-09-13T16:43:56.508720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872216+0000","last_change":"2024-09-13T16:43:54.995262+0000","last_active":"2024-09-13T16:43:56.872216+0000","last_peered":"2024-09-13T16:43:56.872216+0000","last_clean":"2024-09-13T16:43:56.872216+0000","last_became_active":"2024-09-13T16:43:54.994891+0000","last_became_peered":"2024-09-13T16:43:54.994891+0000","last_unstale":"2024-09-13T16:43:56.872216+0000","last_undegraded":"2024-09-13T16:43:56.872216+0000","last_fullsized":"2024-09-13T16:43:56.872216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975286+0000","last_change":"2024-09-13T16:43:54.981534+0000","last_active":"2024-09-13T16:43:55.975286+0000","last_peered":"2024-09-13T16:43:55.975286+0000","last_clean":"2024-09-13T16:43:55.975286+0000","last_became_active":"2024-09-13T16:43:54.981191+0000","last_became_peered":"2024-09-13T16:43:54.981191+0000","last_unstale":"2024-09-13T16:43:55.975286+0000","last_undegraded":"2024-09-13T16:43:55.975286+0000","last_fullsized":"2024-09-13T16:43:55.975286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485880+0000","last_change":"2024-09-13T16:43:54.994157+0000","last_active":"2024-09-13T16:43:55.485880+0000","last_peered":"2024-09-13T16:43:55.485880+0000","last_clean":"2024-09-13T16:43:55.485880+0000","last_became_active":"2024-09-13T16:43:54.993915+0000","last_became_peered":"2024-09-13T16:43:54.993915+0000","last_unstale":"2024-09-13T16:43:55.485880+0000","last_undegraded":"2024-09-13T16:43:55.485880+0000","last_fullsized":"2024-09-13T16:43:55.485880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486053+0000","last_change":"2024-09-13T16:43:54.987328+0000","last_active":"2024-09-13T16:43:55.486053+0000","last_peered":"2024-09-13T16:43:55.486053+0000","last_clean":"2024-09-13T16:43:55.486053+0000","last_became_active":"2024-09-13T16:43:54.987048+0000","last_became_peered":"2024-09-13T16:43:54.987048+0000","last_unstale":"2024-09-13T16:43:55.486053+0000","last_undegraded":"2024-09-13T16:43:55.486053+0000","last_fullsized":"2024-09-13T16:43:55.486053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.979840+0000","last_change":"2024-09-13T16:43:54.985628+0000","last_active":"2024-09-13T16:43:55.979840+0000","last_peered":"2024-09-13T16:43:55.979840+0000","last_clean":"2024-09-13T16:43:55.979840+0000","last_became_active":"2024-09-13T16:43:54.985378+0000","last_became_peered":"2024-09-13T16:43:54.985378+0000","last_unstale":"2024-09-13T16:43:55.979840+0000","last_undegraded":"2024-09-13T16:43:55.979840+0000","last_fullsized":"2024-09-13T16:43:55.979840+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178469+0000","last_change":"2024-09-13T16:43:54.996712+0000","last_active":"2024-09-13T16:43:59.178469+0000","last_peered":"2024-09-13T16:43:59.178469+0000","last_clean":"2024-09-13T16:43:59.178469+0000","last_became_active":"2024-09-13T16:43:54.996523+0000","last_became_peered":"2024-09-13T16:43:54.996523+0000","last_unstale":"2024-09-13T16:43:59.178469+0000","last_undegraded":"2024-09-13T16:43:59.178469+0000","last_fullsized":"2024-09-13T16:43:59.178469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260867+0000","last_change":"2024-09-13T16:43:54.984879+0000","last_active":"2024-09-13T16:43:56.260867+0000","last_peered":"2024-09-13T16:43:56.260867+0000","last_clean":"2024-09-13T16:43:56.260867+0000","last_became_active":"2024-09-13T16:43:54.984595+0000","last_became_peered":"2024-09-13T16:43:54.984595+0000","last_unstale":"2024-09-13T16:43:56.260867+0000","last_undegraded":"2024-09-13T16:43:56.260867+0000","last_fullsized":"2024-09-13T16:43:56.260867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.871914+0000","last_change":"2024-09-13T16:43:54.996457+0000","last_active":"2024-09-13T16:43:56.871914+0000","last_peered":"2024-09-13T16:43:56.871914+0000","last_clean":"2024-09-13T16:43:56.871914+0000","last_became_active":"2024-09-13T16:43:54.996275+0000","last_became_peered":"2024-09-13T16:43:54.996275+0000","last_unstale":"2024-09-13T16:43:56.871914+0000","last_undegraded":"2024-09-13T16:43:56.871914+0000","last_fullsized":"2024-09-13T16:43:56.871914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485929+0000","last_change":"2024-09-13T16:43:54.985368+0000","last_active":"2024-09-13T16:43:55.485929+0000","last_peered":"2024-09-13T16:43:55.485929+0000","last_clean":"2024-09-13T16:43:55.485929+0000","last_became_active":"2024-09-13T16:43:54.985130+0000","last_became_peered":"2024-09-13T16:43:54.985130+0000","last_unstale":"2024-09-13T16:43:55.485929+0000","last_undegraded":"2024-09-13T16:43:55.485929+0000","last_fullsized":"2024-09-13T16:43:55.485929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508531+0000","last_change":"2024-09-13T16:43:54.991862+0000","last_active":"2024-09-13T16:43:56.508531+0000","last_peered":"2024-09-13T16:43:56.508531+0000","last_clean":"2024-09-13T16:43:56.508531+0000","last_became_active":"2024-09-13T16:43:54.991554+0000","last_became_peered":"2024-09-13T16:43:54.991554+0000","last_unstale":"2024-09-13T16:43:56.508531+0000","last_undegraded":"2024-09-13T16:43:56.508531+0000","last_fullsized":"2024-09-13T16:43:56.508531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485896+0000","last_change":"2024-09-13T16:43:54.992241+0000","last_active":"2024-09-13T16:43:55.485896+0000","last_peered":"2024-09-13T16:43:55.485896+0000","last_clean":"2024-09-13T16:43:55.485896+0000","last_became_active":"2024-09-13T16:43:54.991687+0000","last_became_peered":"2024-09-13T16:43:54.991687+0000","last_unstale":"2024-09-13T16:43:55.485896+0000","last_undegraded":"2024-09-13T16:43:55.485896+0000","last_fullsized":"2024-09-13T16:43:55.485896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.509024+0000","last_change":"2024-09-13T16:43:54.991366+0000","last_active":"2024-09-13T16:43:56.509024+0000","last_peered":"2024-09-13T16:43:56.509024+0000","last_clean":"2024-09-13T16:43:56.509024+0000","last_became_active":"2024-09-13T16:43:54.991127+0000","last_became_peered":"2024-09-13T16:43:54.991127+0000","last_unstale":"2024-09-13T16:43:56.509024+0000","last_undegraded":"2024-09-13T16:43:56.509024+0000","last_fullsized":"2024-09-13T16:43:56.509024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.261006+0000","last_change":"2024-09-13T16:43:54.985615+0000","last_active":"2024-09-13T16:43:56.261006+0000","last_peered":"2024-09-13T16:43:56.261006+0000","last_clean":"2024-09-13T16:43:56.261006+0000","last_became_active":"2024-09-13T16:43:54.985299+0000","last_became_peered":"2024-09-13T16:43:54.985299+0000","last_unstale":"2024-09-13T16:43:56.261006+0000","last_undegraded":"2024-09-13T16:43:56.261006+0000","last_fullsized":"2024-09-13T16:43:56.261006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975382+0000","last_change":"2024-09-13T16:43:54.993325+0000","last_active":"2024-09-13T16:43:55.975382+0000","last_peered":"2024-09-13T16:43:55.975382+0000","last_clean":"2024-09-13T16:43:55.975382+0000","last_became_active":"2024-09-13T16:43:54.993009+0000","last_became_peered":"2024-09-13T16:43:54.993009+0000","last_unstale":"2024-09-13T16:43:55.975382+0000","last_undegraded":"2024-09-13T16:43:55.975382+0000","last_fullsized":"2024-09-13T16:43:55.975382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508820+0000","last_change":"2024-09-13T16:43:54.998229+0000","last_active":"2024-09-13T16:43:56.508820+0000","last_peered":"2024-09-13T16:43:56.508820+0000","last_clean":"2024-09-13T16:43:56.508820+0000","last_became_active":"2024-09-13T16:43:54.995774+0000","last_became_peered":"2024-09-13T16:43:54.995774+0000","last_unstale":"2024-09-13T16:43:56.508820+0000","last_undegraded":"2024-09-13T16:43:56.508820+0000","last_fullsized":"2024-09-13T16:43:56.508820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872235+0000","last_change":"2024-09-13T16:43:55.002126+0000","last_active":"2024-09-13T16:43:56.872235+0000","last_peered":"2024-09-13T16:43:56.872235+0000","last_clean":"2024-09-13T16:43:56.872235+0000","last_became_active":"2024-09-13T16:43:55.001713+0000","last_became_peered":"2024-09-13T16:43:55.001713+0000","last_unstale":"2024-09-13T16:43:56.872235+0000","last_undegraded":"2024-09-13T16:43:56.872235+0000","last_fullsized":"2024-09-13T16:43:56.872235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485939+0000","last_change":"2024-09-13T16:43:54.992745+0000","last_active":"2024-09-13T16:43:55.485939+0000","last_peered":"2024-09-13T16:43:55.485939+0000","last_clean":"2024-09-13T16:43:55.485939+0000","last_became_active":"2024-09-13T16:43:54.992508+0000","last_became_peered":"2024-09-13T16:43:54.992508+0000","last_unstale":"2024-09-13T16:43:55.485939+0000","last_undegraded":"2024-09-13T16:43:55.485939+0000","last_fullsized":"2024-09-13T16:43:55.485939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:23.913470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.979793+0000","last_change":"2024-09-13T16:43:54.990631+0000","last_active":"2024-09-13T16:43:55.979793+0000","last_peered":"2024-09-13T16:43:55.979793+0000","last_clean":"2024-09-13T16:43:55.979793+0000","last_became_active":"2024-09-13T16:43:54.986277+0000","last_became_peered":"2024-09-13T16:43:54.986277+0000","last_unstale":"2024-09-13T16:43:55.979793+0000","last_undegraded":"2024-09-13T16:43:55.979793+0000","last_fullsized":"2024-09-13T16:43:55.979793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260674+0000","last_change":"2024-09-13T16:43:55.000059+0000","last_active":"2024-09-13T16:43:56.260674+0000","last_peered":"2024-09-13T16:43:56.260674+0000","last_clean":"2024-09-13T16:43:56.260674+0000","last_became_active":"2024-09-13T16:43:54.999778+0000","last_became_peered":"2024-09-13T16:43:54.999778+0000","last_unstale":"2024-09-13T16:43:56.260674+0000","last_undegraded":"2024-09-13T16:43:56.260674+0000","last_fullsized":"2024-09-13T16:43:56.260674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935551+0000","last_change":"2024-09-13T16:43:54.988375+0000","last_active":"2024-09-13T16:43:56.935551+0000","last_peered":"2024-09-13T16:43:56.935551+0000","last_clean":"2024-09-13T16:43:56.935551+0000","last_became_active":"2024-09-13T16:43:54.988131+0000","last_became_peered":"2024-09-13T16:43:54.988131+0000","last_unstale":"2024-09-13T16:43:56.935551+0000","last_undegraded":"2024-09-13T16:43:56.935551+0000","last_fullsized":"2024-09-13T16:43:56.935551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.635755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:09.023856+0000","last_change":"2024-09-13T16:43:54.555042+0000","last_active":"2024-09-13T16:44:09.023856+0000","last_peered":"2024-09-13T16:44:09.023856+0000","last_clean":"2024-09-13T16:44:09.023856+0000","last_became_active":"2024-09-13T16:43:54.554697+0000","last_became_peered":"2024-09-13T16:43:54.554697+0000","last_unstale":"2024-09-13T16:44:09.023856+0000","last_undegraded":"2024-09-13T16:44:09.023856+0000","last_fullsized":"2024-09-13T16:44:09.023856+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872075+0000","last_change":"2024-09-13T16:43:54.998755+0000","last_active":"2024-09-13T16:43:56.872075+0000","last_peered":"2024-09-13T16:43:56.872075+0000","last_clean":"2024-09-13T16:43:56.872075+0000","last_became_active":"2024-09-13T16:43:54.998399+0000","last_became_peered":"2024-09-13T16:43:54.998399+0000","last_unstale":"2024-09-13T16:43:56.872075+0000","last_undegraded":"2024-09-13T16:43:56.872075+0000","last_fullsized":"2024-09-13T16:43:56.872075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.675242+0000","last_change":"2024-09-13T16:43:54.553250+0000","last_active":"2024-09-13T16:44:19.675242+0000","last_peered":"2024-09-13T16:44:19.675242+0000","last_clean":"2024-09-13T16:44:19.675242+0000","last_became_active":"2024-09-13T16:43:54.552977+0000","last_became_peered":"2024-09-13T16:43:54.552977+0000","last_unstale":"2024-09-13T16:44:19.675242+0000","last_undegraded":"2024-09-13T16:44:19.675242+0000","last_fullsized":"2024-09-13T16:44:19.675242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:14.094591+0000","last_change":"2024-09-13T16:43:54.558065+0000","last_active":"2024-09-13T16:44:14.094591+0000","last_peered":"2024-09-13T16:44:14.094591+0000","last_clean":"2024-09-13T16:44:14.094591+0000","last_became_active":"2024-09-13T16:43:54.557722+0000","last_became_peered":"2024-09-13T16:43:54.557722+0000","last_unstale":"2024-09-13T16:44:14.094591+0000","last_undegraded":"2024-09-13T16:44:14.094591+0000","last_fullsized":"2024-09-13T16:44:14.094591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508519+0000","last_change":"2024-09-13T16:43:55.002041+0000","last_active":"2024-09-13T16:43:56.508519+0000","last_peered":"2024-09-13T16:43:56.508519+0000","last_clean":"2024-09-13T16:43:56.508519+0000","last_became_active":"2024-09-13T16:43:55.001512+0000","last_became_peered":"2024-09-13T16:43:55.001512+0000","last_unstale":"2024-09-13T16:43:56.508519+0000","last_undegraded":"2024-09-13T16:43:56.508519+0000","last_fullsized":"2024-09-13T16:43:56.508519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.680255+0000","last_change":"2024-09-13T16:43:54.550389+0000","last_active":"2024-09-13T16:44:19.680255+0000","last_peered":"2024-09-13T16:44:19.680255+0000","last_clean":"2024-09-13T16:44:19.680255+0000","last_became_active":"2024-09-13T16:43:54.550154+0000","last_became_peered":"2024-09-13T16:43:54.550154+0000","last_unstale":"2024-09-13T16:44:19.680255+0000","last_undegraded":"2024-09-13T16:44:19.680255+0000","last_fullsized":"2024-09-13T16:44:19.680255+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975641+0000","last_change":"2024-09-13T16:43:54.993377+0000","last_active":"2024-09-13T16:43:55.975641+0000","last_peered":"2024-09-13T16:43:55.975641+0000","last_clean":"2024-09-13T16:43:55.975641+0000","last_became_active":"2024-09-13T16:43:54.993036+0000","last_became_peered":"2024-09-13T16:43:54.993036+0000","last_unstale":"2024-09-13T16:43:55.975641+0000","last_undegraded":"2024-09-13T16:43:55.975641+0000","last_fullsized":"2024-09-13T16:43:55.975641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:09.022198+0000","last_change":"2024-09-13T16:43:54.555528+0000","last_active":"2024-09-13T16:44:09.022198+0000","last_peered":"2024-09-13T16:44:09.022198+0000","last_clean":"2024-09-13T16:44:09.022198+0000","last_became_active":"2024-09-13T16:43:54.555267+0000","last_became_peered":"2024-09-13T16:43:54.555267+0000","last_unstale":"2024-09-13T16:44:09.022198+0000","last_undegraded":"2024-09-13T16:44:09.022198+0000","last_fullsized":"2024-09-13T16:44:09.022198+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486032+0000","last_change":"2024-09-13T16:43:54.981706+0000","last_active":"2024-09-13T16:43:55.486032+0000","last_peered":"2024-09-13T16:43:55.486032+0000","last_clean":"2024-09-13T16:43:55.486032+0000","last_became_active":"2024-09-13T16:43:54.981429+0000","last_became_peered":"2024-09-13T16:43:54.981429+0000","last_unstale":"2024-09-13T16:43:55.486032+0000","last_undegraded":"2024-09-13T16:43:55.486032+0000","last_fullsized":"2024-09-13T16:43:55.486032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.680087+0000","last_change":"2024-09-13T16:43:54.554593+0000","last_active":"2024-09-13T16:44:25.680087+0000","last_peered":"2024-09-13T16:44:25.680087+0000","last_clean":"2024-09-13T16:44:25.680087+0000","last_became_active":"2024-09-13T16:43:54.554307+0000","last_became_peered":"2024-09-13T16:43:54.554307+0000","last_unstale":"2024-09-13T16:44:25.680087+0000","last_undegraded":"2024-09-13T16:44:25.680087+0000","last_fullsized":"2024-09-13T16:44:25.680087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872139+0000","last_change":"2024-09-13T16:43:54.993485+0000","last_active":"2024-09-13T16:43:56.872139+0000","last_peered":"2024-09-13T16:43:56.872139+0000","last_clean":"2024-09-13T16:43:56.872139+0000","last_became_active":"2024-09-13T16:43:54.993226+0000","last_became_peered":"2024-09-13T16:43:54.993226+0000","last_unstale":"2024-09-13T16:43:56.872139+0000","last_undegraded":"2024-09-13T16:43:56.872139+0000","last_fullsized":"2024-09-13T16:43:56.872139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:09.023221+0000","last_change":"2024-09-13T16:43:54.562976+0000","last_active":"2024-09-13T16:44:09.023221+0000","last_peered":"2024-09-13T16:44:09.023221+0000","last_clean":"2024-09-13T16:44:09.023221+0000","last_became_active":"2024-09-13T16:43:54.560480+0000","last_became_peered":"2024-09-13T16:43:54.560480+0000","last_unstale":"2024-09-13T16:44:09.023221+0000","last_undegraded":"2024-09-13T16:44:09.023221+0000","last_fullsized":"2024-09-13T16:44:09.023221+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935689+0000","last_change":"2024-09-13T16:43:54.986459+0000","last_active":"2024-09-13T16:43:56.935689+0000","last_peered":"2024-09-13T16:43:56.935689+0000","last_clean":"2024-09-13T16:43:56.935689+0000","last_became_active":"2024-09-13T16:43:54.986083+0000","last_became_peered":"2024-09-13T16:43:54.986083+0000","last_unstale":"2024-09-13T16:43:56.935689+0000","last_undegraded":"2024-09-13T16:43:56.935689+0000","last_fullsized":"2024-09-13T16:43:56.935689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:51.960459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'10","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.678669+0000","last_change":"2024-09-13T16:43:54.565929+0000","last_active":"2024-09-13T16:44:25.678669+0000","last_peered":"2024-09-13T16:44:25.678669+0000","last_clean":"2024-09-13T16:44:25.678669+0000","last_became_active":"2024-09-13T16:43:54.565701+0000","last_became_peered":"2024-09-13T16:43:54.565701+0000","last_unstale":"2024-09-13T16:44:25.678669+0000","last_undegraded":"2024-09-13T16:44:25.678669+0000","last_fullsized":"2024-09-13T16:44:25.678669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178232+0000","last_change":"2024-09-13T16:43:54.989068+0000","last_active":"2024-09-13T16:43:59.178232+0000","last_peered":"2024-09-13T16:43:59.178232+0000","last_clean":"2024-09-13T16:43:59.178232+0000","last_became_active":"2024-09-13T16:43:54.988779+0000","last_became_peered":"2024-09-13T16:43:54.988779+0000","last_unstale":"2024-09-13T16:43:59.178232+0000","last_undegraded":"2024-09-13T16:43:59.178232+0000","last_fullsized":"2024-09-13T16:43:59.178232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.682164+0000","last_change":"2024-09-13T16:43:54.561487+0000","last_active":"2024-09-13T16:44:25.682164+0000","last_peered":"2024-09-13T16:44:25.682164+0000","last_clean":"2024-09-13T16:44:25.682164+0000","last_became_active":"2024-09-13T16:43:54.561077+0000","last_became_peered":"2024-09-13T16:43:54.561077+0000","last_unstale":"2024-09-13T16:44:25.682164+0000","last_undegraded":"2024-09-13T16:44:25.682164+0000","last_fullsized":"2024-09-13T16:44:25.682164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872141+0000","last_change":"2024-09-13T16:43:54.999517+0000","last_active":"2024-09-13T16:43:56.872141+0000","last_peered":"2024-09-13T16:43:56.872141+0000","last_clean":"2024-09-13T16:43:56.872141+0000","last_became_active":"2024-09-13T16:43:54.999283+0000","last_became_peered":"2024-09-13T16:43:54.999283+0000","last_unstale":"2024-09-13T16:43:56.872141+0000","last_undegraded":"2024-09-13T16:43:56.872141+0000","last_fullsized":"2024-09-13T16:43:56.872141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:20.207775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.674779+0000","last_change":"2024-09-13T16:43:54.542871+0000","last_active":"2024-09-13T16:44:19.674779+0000","last_peered":"2024-09-13T16:44:19.674779+0000","last_clean":"2024-09-13T16:44:19.674779+0000","last_became_active":"2024-09-13T16:43:54.542548+0000","last_became_peered":"2024-09-13T16:43:54.542548+0000","last_unstale":"2024-09-13T16:44:19.674779+0000","last_undegraded":"2024-09-13T16:44:19.674779+0000","last_fullsized":"2024-09-13T16:44:19.674779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178181+0000","last_change":"2024-09-13T16:43:54.989168+0000","last_active":"2024-09-13T16:43:59.178181+0000","last_peered":"2024-09-13T16:43:59.178181+0000","last_clean":"2024-09-13T16:43:59.178181+0000","last_became_active":"2024-09-13T16:43:54.988716+0000","last_became_peered":"2024-09-13T16:43:54.988716+0000","last_unstale":"2024-09-13T16:43:59.178181+0000","last_undegraded":"2024-09-13T16:43:59.178181+0000","last_fullsized":"2024-09-13T16:43:59.178181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.670560+0000","last_change":"2024-09-13T16:43:54.551551+0000","last_active":"2024-09-13T16:44:19.670560+0000","last_peered":"2024-09-13T16:44:19.670560+0000","last_clean":"2024-09-13T16:44:19.670560+0000","last_became_active":"2024-09-13T16:43:54.551293+0000","last_became_peered":"2024-09-13T16:43:54.551293+0000","last_unstale":"2024-09-13T16:44:19.670560+0000","last_undegraded":"2024-09-13T16:44:19.670560+0000","last_fullsized":"2024-09-13T16:44:19.670560+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260677+0000","last_change":"2024-09-13T16:43:54.994516+0000","last_active":"2024-09-13T16:43:56.260677+0000","last_peered":"2024-09-13T16:43:56.260677+0000","last_clean":"2024-09-13T16:43:56.260677+0000","last_became_active":"2024-09-13T16:43:54.994245+0000","last_became_peered":"2024-09-13T16:43:54.994245+0000","last_unstale":"2024-09-13T16:43:56.260677+0000","last_undegraded":"2024-09-13T16:43:56.260677+0000","last_fullsized":"2024-09-13T16:43:56.260677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":28,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.673680+0000","last_change":"2024-09-13T16:43:54.562232+0000","last_active":"2024-09-13T16:44:19.673680+0000","last_peered":"2024-09-13T16:44:19.673680+0000","last_clean":"2024-09-13T16:44:19.673680+0000","last_became_active":"2024-09-13T16:43:54.561939+0000","last_became_peered":"2024-09-13T16:43:54.561939+0000","last_unstale":"2024-09-13T16:44:19.673680+0000","last_undegraded":"2024-09-13T16:44:19.673680+0000","last_fullsized":"2024-09-13T16:44:19.673680+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178047+0000","last_change":"2024-09-13T16:43:54.988439+0000","last_active":"2024-09-13T16:43:59.178047+0000","last_peered":"2024-09-13T16:43:59.178047+0000","last_clean":"2024-09-13T16:43:59.178047+0000","last_became_active":"2024-09-13T16:43:54.988172+0000","last_became_peered":"2024-09-13T16:43:54.988172+0000","last_unstale":"2024-09-13T16:43:59.178047+0000","last_undegraded":"2024-09-13T16:43:59.178047+0000","last_fullsized":"2024-09-13T16:43:59.178047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.986952+0000","last_change":"2024-09-13T16:43:54.559372+0000","last_active":"2024-09-13T16:43:59.986952+0000","last_peered":"2024-09-13T16:43:59.986952+0000","last_clean":"2024-09-13T16:43:59.986952+0000","last_became_active":"2024-09-13T16:43:54.559111+0000","last_became_peered":"2024-09-13T16:43:54.559111+0000","last_unstale":"2024-09-13T16:43:59.986952+0000","last_undegraded":"2024-09-13T16:43:59.986952+0000","last_fullsized":"2024-09-13T16:43:59.986952+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:40.399883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178451+0000","last_change":"2024-09-13T16:43:54.988313+0000","last_active":"2024-09-13T16:43:59.178451+0000","last_peered":"2024-09-13T16:43:59.178451+0000","last_clean":"2024-09-13T16:43:59.178451+0000","last_became_active":"2024-09-13T16:43:54.988059+0000","last_became_peered":"2024-09-13T16:43:54.988059+0000","last_unstale":"2024-09-13T16:43:59.178451+0000","last_undegraded":"2024-09-13T16:43:59.178451+0000","last_fullsized":"2024-09-13T16:43:59.178451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.678704+0000","last_change":"2024-09-13T16:43:54.566410+0000","last_active":"2024-09-13T16:44:25.678704+0000","last_peered":"2024-09-13T16:44:25.678704+0000","last_clean":"2024-09-13T16:44:25.678704+0000","last_became_active":"2024-09-13T16:43:54.565861+0000","last_became_peered":"2024-09-13T16:43:54.565861+0000","last_unstale":"2024-09-13T16:44:25.678704+0000","last_undegraded":"2024-09-13T16:44:25.678704+0000","last_fullsized":"2024-09-13T16:44:25.678704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178547+0000","last_change":"2024-09-13T16:43:54.989506+0000","last_active":"2024-09-13T16:43:59.178547+0000","last_peered":"2024-09-13T16:43:59.178547+0000","last_clean":"2024-09-13T16:43:59.178547+0000","last_became_active":"2024-09-13T16:43:54.989233+0000","last_became_peered":"2024-09-13T16:43:54.989233+0000","last_unstale":"2024-09-13T16:43:59.178547+0000","last_undegraded":"2024-09-13T16:43:59.178547+0000","last_fullsized":"2024-09-13T16:43:59.178547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:23.913470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.675151+0000","last_change":"2024-09-13T16:43:54.559723+0000","last_active":"2024-09-13T16:44:19.675151+0000","last_peered":"2024-09-13T16:44:19.675151+0000","last_clean":"2024-09-13T16:44:19.675151+0000","last_became_active":"2024-09-13T16:43:54.559353+0000","last_became_peered":"2024-09-13T16:43:54.559353+0000","last_unstale":"2024-09-13T16:44:19.675151+0000","last_undegraded":"2024-09-13T16:44:19.675151+0000","last_fullsized":"2024-09-13T16:44:19.675151+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486000+0000","last_change":"2024-09-13T16:43:54.992511+0000","last_active":"2024-09-13T16:43:55.486000+0000","last_peered":"2024-09-13T16:43:55.486000+0000","last_clean":"2024-09-13T16:43:55.486000+0000","last_became_active":"2024-09-13T16:43:54.992249+0000","last_became_peered":"2024-09-13T16:43:54.992249+0000","last_unstale":"2024-09-13T16:43:55.486000+0000","last_undegraded":"2024-09-13T16:43:55.486000+0000","last_fullsized":"2024-09-13T16:43:55.486000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.675039+0000","last_change":"2024-09-13T16:43:54.551390+0000","last_active":"2024-09-13T16:44:19.675039+0000","last_peered":"2024-09-13T16:44:19.675039+0000","last_clean":"2024-09-13T16:44:19.675039+0000","last_became_active":"2024-09-13T16:43:54.547074+0000","last_became_peered":"2024-09-13T16:43:54.547074+0000","last_unstale":"2024-09-13T16:44:19.675039+0000","last_undegraded":"2024-09-13T16:44:19.675039+0000","last_fullsized":"2024-09-13T16:44:19.675039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.178318+0000","last_change":"2024-09-13T16:43:54.990458+0000","last_active":"2024-09-13T16:43:59.178318+0000","last_peered":"2024-09-13T16:43:59.178318+0000","last_clean":"2024-09-13T16:43:59.178318+0000","last_became_active":"2024-09-13T16:43:54.990196+0000","last_became_peered":"2024-09-13T16:43:54.990196+0000","last_unstale":"2024-09-13T16:43:59.178318+0000","last_undegraded":"2024-09-13T16:43:59.178318+0000","last_fullsized":"2024-09-13T16:43:59.178318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:54.981787+0000","last_change":"2024-09-13T16:43:51.914639+0000","last_active":"2024-09-13T16:43:54.981787+0000","last_peered":"2024-09-13T16:43:54.981787+0000","last_clean":"2024-09-13T16:43:54.981787+0000","last_became_active":"2024-09-13T16:43:51.913797+0000","last_became_peered":"2024-09-13T16:43:51.913797+0000","last_unstale":"2024-09-13T16:43:54.981787+0000","last_undegraded":"2024-09-13T16:43:54.981787+0000","last_fullsized":"2024-09-13T16:43:54.981787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_clean_scrub_stamp":"2024-09-13T16:43:50.498116+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:12.730911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.672667+0000","last_change":"2024-09-13T16:43:54.564153+0000","last_active":"2024-09-13T16:44:19.672667+0000","last_peered":"2024-09-13T16:44:19.672667+0000","last_clean":"2024-09-13T16:44:19.672667+0000","last_became_active":"2024-09-13T16:43:54.563949+0000","last_became_peered":"2024-09-13T16:43:54.563949+0000","last_unstale":"2024-09-13T16:44:19.672667+0000","last_undegraded":"2024-09-13T16:44:19.672667+0000","last_fullsized":"2024-09-13T16:44:19.672667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508836+0000","last_change":"2024-09-13T16:43:54.997576+0000","last_active":"2024-09-13T16:43:56.508836+0000","last_peered":"2024-09-13T16:43:56.508836+0000","last_clean":"2024-09-13T16:43:56.508836+0000","last_became_active":"2024-09-13T16:43:54.997315+0000","last_became_peered":"2024-09-13T16:43:54.997315+0000","last_unstale":"2024-09-13T16:43:56.508836+0000","last_undegraded":"2024-09-13T16:43:56.508836+0000","last_fullsized":"2024-09-13T16:43:56.508836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:17.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:14.094371+0000","last_change":"2024-09-13T16:43:54.564804+0000","last_active":"2024-09-13T16:44:14.094371+0000","last_peered":"2024-09-13T16:44:14.094371+0000","last_clean":"2024-09-13T16:44:14.094371+0000","last_became_active":"2024-09-13T16:43:54.564605+0000","last_became_peered":"2024-09-13T16:43:54.564605+0000","last_unstale":"2024-09-13T16:44:14.094371+0000","last_undegraded":"2024-09-13T16:44:14.094371+0000","last_fullsized":"2024-09-13T16:44:14.094371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.871801+0000","last_change":"2024-09-13T16:43:54.997884+0000","last_active":"2024-09-13T16:43:56.871801+0000","last_peered":"2024-09-13T16:43:56.871801+0000","last_clean":"2024-09-13T16:43:56.871801+0000","last_became_active":"2024-09-13T16:43:54.997510+0000","last_became_peered":"2024-09-13T16:43:54.997510+0000","last_unstale":"2024-09-13T16:43:56.871801+0000","last_undegraded":"2024-09-13T16:43:56.871801+0000","last_fullsized":"2024-09-13T16:43:56.871801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:14.094845+0000","last_change":"2024-09-13T16:43:54.566294+0000","last_active":"2024-09-13T16:44:14.094845+0000","last_peered":"2024-09-13T16:44:14.094845+0000","last_clean":"2024-09-13T16:44:14.094845+0000","last_became_active":"2024-09-13T16:43:54.566023+0000","last_became_peered":"2024-09-13T16:43:54.566023+0000","last_unstale":"2024-09-13T16:44:14.094845+0000","last_undegraded":"2024-09-13T16:44:14.094845+0000","last_fullsized":"2024-09-13T16:44:14.094845+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975316+0000","last_change":"2024-09-13T16:43:54.981116+0000","last_active":"2024-09-13T16:43:55.975316+0000","last_peered":"2024-09-13T16:43:55.975316+0000","last_clean":"2024-09-13T16:43:55.975316+0000","last_became_active":"2024-09-13T16:43:54.980766+0000","last_became_peered":"2024-09-13T16:43:54.980766+0000","last_unstale":"2024-09-13T16:43:55.975316+0000","last_undegraded":"2024-09-13T16:43:55.975316+0000","last_fullsized":"2024-09-13T16:43:55.975316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.682397+0000","last_change":"2024-09-13T16:43:54.563716+0000","last_active":"2024-09-13T16:44:25.682397+0000","last_peered":"2024-09-13T16:44:25.682397+0000","last_clean":"2024-09-13T16:44:25.682397+0000","last_became_active":"2024-09-13T16:43:54.563522+0000","last_became_peered":"2024-09-13T16:43:54.563522+0000","last_unstale":"2024-09-13T16:44:25.682397+0000","last_undegraded":"2024-09-13T16:44:25.682397+0000","last_fullsized":"2024-09-13T16:44:25.682397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935617+0000","last_change":"2024-09-13T16:43:54.999051+0000","last_active":"2024-09-13T16:43:56.935617+0000","last_peered":"2024-09-13T16:43:56.935617+0000","last_clean":"2024-09-13T16:43:56.935617+0000","last_became_active":"2024-09-13T16:43:54.998803+0000","last_became_peered":"2024-09-13T16:43:54.998803+0000","last_unstale":"2024-09-13T16:43:56.935617+0000","last_undegraded":"2024-09-13T16:43:56.935617+0000","last_fullsized":"2024-09-13T16:43:56.935617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975558+0000","last_change":"2024-09-13T16:43:54.569945+0000","last_active":"2024-09-13T16:43:55.975558+0000","last_peered":"2024-09-13T16:43:55.975558+0000","last_clean":"2024-09-13T16:43:55.975558+0000","last_became_active":"2024-09-13T16:43:54.568846+0000","last_became_peered":"2024-09-13T16:43:54.568846+0000","last_unstale":"2024-09-13T16:43:55.975558+0000","last_undegraded":"2024-09-13T16:43:55.975558+0000","last_fullsized":"2024-09-13T16:43:55.975558+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486021+0000","last_change":"2024-09-13T16:43:54.992082+0000","last_active":"2024-09-13T16:43:55.486021+0000","last_peered":"2024-09-13T16:43:55.486021+0000","last_clean":"2024-09-13T16:43:55.486021+0000","last_became_active":"2024-09-13T16:43:54.991659+0000","last_became_peered":"2024-09-13T16:43:54.991659+0000","last_unstale":"2024-09-13T16:43:55.486021+0000","last_undegraded":"2024-09-13T16:43:55.486021+0000","last_fullsized":"2024-09-13T16:43:55.486021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.676205+0000","last_change":"2024-09-13T16:43:54.569111+0000","last_active":"2024-09-13T16:44:19.676205+0000","last_peered":"2024-09-13T16:44:19.676205+0000","last_clean":"2024-09-13T16:44:19.676205+0000","last_became_active":"2024-09-13T16:43:54.568903+0000","last_became_peered":"2024-09-13T16:43:54.568903+0000","last_unstale":"2024-09-13T16:44:19.676205+0000","last_undegraded":"2024-09-13T16:44:19.676205+0000","last_fullsized":"2024-09-13T16:44:19.676205+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.975475+0000","last_change":"2024-09-13T16:43:54.980100+0000","last_active":"2024-09-13T16:43:55.975475+0000","last_peered":"2024-09-13T16:43:55.975475+0000","last_clean":"2024-09-13T16:43:55.975475+0000","last_became_active":"2024-09-13T16:43:54.979584+0000","last_became_peered":"2024-09-13T16:43:54.979584+0000","last_unstale":"2024-09-13T16:43:55.975475+0000","last_undegraded":"2024-09-13T16:43:55.975475+0000","last_fullsized":"2024-09-13T16:43:55.975475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'7","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.673990+0000","last_change":"2024-09-13T16:43:54.560710+0000","last_active":"2024-09-13T16:44:19.673990+0000","last_peered":"2024-09-13T16:44:19.673990+0000","last_clean":"2024-09-13T16:44:19.673990+0000","last_became_active":"2024-09-13T16:43:54.560030+0000","last_became_peered":"2024-09-13T16:43:54.560030+0000","last_unstale":"2024-09-13T16:44:19.673990+0000","last_undegraded":"2024-09-13T16:44:19.673990+0000","last_fullsized":"2024-09-13T16:44:19.673990+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.509048+0000","last_change":"2024-09-13T16:43:54.990812+0000","last_active":"2024-09-13T16:43:56.509048+0000","last_peered":"2024-09-13T16:43:56.509048+0000","last_clean":"2024-09-13T16:43:56.509048+0000","last_became_active":"2024-09-13T16:43:54.990440+0000","last_became_peered":"2024-09-13T16:43:54.990440+0000","last_unstale":"2024-09-13T16:43:56.509048+0000","last_undegraded":"2024-09-13T16:43:56.509048+0000","last_fullsized":"2024-09-13T16:43:56.509048+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.682745+0000","last_change":"2024-09-13T16:43:54.558439+0000","last_active":"2024-09-13T16:44:25.682745+0000","last_peered":"2024-09-13T16:44:25.682745+0000","last_clean":"2024-09-13T16:44:25.682745+0000","last_became_active":"2024-09-13T16:43:54.558194+0000","last_became_peered":"2024-09-13T16:43:54.558194+0000","last_unstale":"2024-09-13T16:44:25.682745+0000","last_undegraded":"2024-09-13T16:44:25.682745+0000","last_fullsized":"2024-09-13T16:44:25.682745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872087+0000","last_change":"2024-09-13T16:43:54.995534+0000","last_active":"2024-09-13T16:43:56.872087+0000","last_peered":"2024-09-13T16:43:56.872087+0000","last_clean":"2024-09-13T16:43:56.872087+0000","last_became_active":"2024-09-13T16:43:54.995397+0000","last_became_peered":"2024-09-13T16:43:54.995397+0000","last_unstale":"2024-09-13T16:43:56.872087+0000","last_undegraded":"2024-09-13T16:43:56.872087+0000","last_fullsized":"2024-09-13T16:43:56.872087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:19.697247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:09.021581+0000","last_change":"2024-09-13T16:43:54.569085+0000","last_active":"2024-09-13T16:44:09.021581+0000","last_peered":"2024-09-13T16:44:09.021581+0000","last_clean":"2024-09-13T16:44:09.021581+0000","last_became_active":"2024-09-13T16:43:54.568892+0000","last_became_peered":"2024-09-13T16:43:54.568892+0000","last_unstale":"2024-09-13T16:44:09.021581+0000","last_undegraded":"2024-09-13T16:44:09.021581+0000","last_fullsized":"2024-09-13T16:44:09.021581+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.935459+0000","last_change":"2024-09-13T16:43:54.984120+0000","last_active":"2024-09-13T16:43:56.935459+0000","last_peered":"2024-09-13T16:43:56.935459+0000","last_clean":"2024-09-13T16:43:56.935459+0000","last_became_active":"2024-09-13T16:43:54.983886+0000","last_became_peered":"2024-09-13T16:43:54.983886+0000","last_unstale":"2024-09-13T16:43:56.935459+0000","last_undegraded":"2024-09-13T16:43:56.935459+0000","last_fullsized":"2024-09-13T16:43:56.935459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.679742+0000","last_change":"2024-09-13T16:43:54.555092+0000","last_active":"2024-09-13T16:44:25.679742+0000","last_peered":"2024-09-13T16:44:25.679742+0000","last_clean":"2024-09-13T16:44:25.679742+0000","last_became_active":"2024-09-13T16:43:54.554230+0000","last_became_peered":"2024-09-13T16:43:54.554230+0000","last_unstale":"2024-09-13T16:44:25.679742+0000","last_undegraded":"2024-09-13T16:44:25.679742+0000","last_fullsized":"2024-09-13T16:44:25.679742+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508681+0000","last_change":"2024-09-13T16:43:54.984588+0000","last_active":"2024-09-13T16:43:56.508681+0000","last_peered":"2024-09-13T16:43:56.508681+0000","last_clean":"2024-09-13T16:43:56.508681+0000","last_became_active":"2024-09-13T16:43:54.984221+0000","last_became_peered":"2024-09-13T16:43:54.984221+0000","last_unstale":"2024-09-13T16:43:56.508681+0000","last_undegraded":"2024-09-13T16:43:56.508681+0000","last_fullsized":"2024-09-13T16:43:56.508681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:19.673474+0000","last_change":"2024-09-13T16:43:54.562646+0000","last_active":"2024-09-13T16:44:19.673474+0000","last_peered":"2024-09-13T16:44:19.673474+0000","last_clean":"2024-09-13T16:44:19.673474+0000","last_became_active":"2024-09-13T16:43:54.556888+0000","last_became_peered":"2024-09-13T16:43:54.556888+0000","last_unstale":"2024-09-13T16:44:19.673474+0000","last_undegraded":"2024-09-13T16:44:19.673474+0000","last_fullsized":"2024-09-13T16:44:19.673474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508792+0000","last_change":"2024-09-13T16:43:54.999328+0000","last_active":"2024-09-13T16:43:56.508792+0000","last_peered":"2024-09-13T16:43:56.508792+0000","last_clean":"2024-09-13T16:43:56.508792+0000","last_became_active":"2024-09-13T16:43:54.999101+0000","last_became_peered":"2024-09-13T16:43:54.999101+0000","last_unstale":"2024-09-13T16:43:56.508792+0000","last_undegraded":"2024-09-13T16:43:56.508792+0000","last_fullsized":"2024-09-13T16:43:56.508792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:09.021554+0000","last_change":"2024-09-13T16:43:54.559214+0000","last_active":"2024-09-13T16:44:09.021554+0000","last_peered":"2024-09-13T16:44:09.021554+0000","last_clean":"2024-09-13T16:44:09.021554+0000","last_became_active":"2024-09-13T16:43:54.558233+0000","last_became_peered":"2024-09-13T16:43:54.558233+0000","last_unstale":"2024-09-13T16:44:09.021554+0000","last_undegraded":"2024-09-13T16:44:09.021554+0000","last_fullsized":"2024-09-13T16:44:09.021554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.260789+0000","last_change":"2024-09-13T16:43:54.986565+0000","last_active":"2024-09-13T16:43:56.260789+0000","last_peered":"2024-09-13T16:43:56.260789+0000","last_clean":"2024-09-13T16:43:56.260789+0000","last_became_active":"2024-09-13T16:43:54.986084+0000","last_became_peered":"2024-09-13T16:43:54.986084+0000","last_unstale":"2024-09-13T16:43:56.260789+0000","last_undegraded":"2024-09-13T16:43:56.260789+0000","last_fullsized":"2024-09-13T16:43:56.260789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.486010+0000","last_change":"2024-09-13T16:43:54.982611+0000","last_active":"2024-09-13T16:43:55.486010+0000","last_peered":"2024-09-13T16:43:55.486010+0000","last_clean":"2024-09-13T16:43:55.486010+0000","last_became_active":"2024-09-13T16:43:54.982338+0000","last_became_peered":"2024-09-13T16:43:54.982338+0000","last_unstale":"2024-09-13T16:43:55.486010+0000","last_undegraded":"2024-09-13T16:43:55.486010+0000","last_fullsized":"2024-09-13T16:43:55.486010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.682678+0000","last_change":"2024-09-13T16:43:54.558799+0000","last_active":"2024-09-13T16:44:25.682678+0000","last_peered":"2024-09-13T16:44:25.682678+0000","last_clean":"2024-09-13T16:44:25.682678+0000","last_became_active":"2024-09-13T16:43:54.558332+0000","last_became_peered":"2024-09-13T16:43:54.558332+0000","last_unstale":"2024-09-13T16:44:25.682678+0000","last_undegraded":"2024-09-13T16:44:25.682678+0000","last_fullsized":"2024-09-13T16:44:25.682678+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:19.201506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T16:43:55.485991+0000","last_change":"2024-09-13T16:43:54.984150+0000","last_active":"2024-09-13T16:43:55.485991+0000","last_peered":"2024-09-13T16:43:55.485991+0000","last_clean":"2024-09-13T16:43:55.485991+0000","last_became_active":"2024-09-13T16:43:54.983881+0000","last_became_peered":"2024-09-13T16:43:54.983881+0000","last_unstale":"2024-09-13T16:43:55.485991+0000","last_undegraded":"2024-09-13T16:43:55.485991+0000","last_fullsized":"2024-09-13T16:43:55.485991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:29.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'12","reported_seq":23,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.682774+0000","last_change":"2024-09-13T16:43:54.559551+0000","last_active":"2024-09-13T16:44:25.682774+0000","last_peered":"2024-09-13T16:44:25.682774+0000","last_clean":"2024-09-13T16:44:25.682774+0000","last_became_active":"2024-09-13T16:43:54.559290+0000","last_became_peered":"2024-09-13T16:43:54.559290+0000","last_unstale":"2024-09-13T16:44:25.682774+0000","last_undegraded":"2024-09-13T16:44:25.682774+0000","last_fullsized":"2024-09-13T16:44:25.682774+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.872084+0000","last_change":"2024-09-13T16:43:55.000783+0000","last_active":"2024-09-13T16:43:56.872084+0000","last_peered":"2024-09-13T16:43:56.872084+0000","last_clean":"2024-09-13T16:43:56.872084+0000","last_became_active":"2024-09-13T16:43:55.000196+0000","last_became_peered":"2024-09-13T16:43:55.000196+0000","last_unstale":"2024-09-13T16:43:56.872084+0000","last_undegraded":"2024-09-13T16:43:56.872084+0000","last_fullsized":"2024-09-13T16:43:56.872084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:59.986199+0000","last_change":"2024-09-13T16:43:54.559886+0000","last_active":"2024-09-13T16:43:59.986199+0000","last_peered":"2024-09-13T16:43:59.986199+0000","last_clean":"2024-09-13T16:43:59.986199+0000","last_became_active":"2024-09-13T16:43:54.559586+0000","last_became_peered":"2024-09-13T16:43:54.559586+0000","last_unstale":"2024-09-13T16:43:59.986199+0000","last_undegraded":"2024-09-13T16:43:59.986199+0000","last_fullsized":"2024-09-13T16:43:59.986199+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:43:56.508978+0000","last_change":"2024-09-13T16:43:54.996099+0000","last_active":"2024-09-13T16:43:56.508978+0000","last_peered":"2024-09-13T16:43:56.508978+0000","last_clean":"2024-09-13T16:43:56.508978+0000","last_became_active":"2024-09-13T16:43:54.995802+0000","last_became_peered":"2024-09-13T16:43:54.995802+0000","last_unstale":"2024-09-13T16:43:56.508978+0000","last_undegraded":"2024-09-13T16:43:56.508978+0000","last_fullsized":"2024-09-13T16:43:56.508978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":24,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T16:44:25.679752+0000","last_change":"2024-09-13T16:43:54.562699+0000","last_active":"2024-09-13T16:44:25.679752+0000","last_peered":"2024-09-13T16:44:25.679752+0000","last_clean":"2024-09-13T16:44:25.679752+0000","last_became_active":"2024-09-13T16:43:54.560552+0000","last_became_peered":"2024-09-13T16:43:54.560552+0000","last_unstale":"2024-09-13T16:44:25.679752+0000","last_undegraded":"2024-09-13T16:44:25.679752+0000","last_fullsized":"2024-09-13T16:44:25.679752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:11.032382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7794,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":15588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":307200,"data_stored":663318,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":73204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":74120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":72180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":73238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":71698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":72382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":294912,"data_stored":661121,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:44:38.790 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-13T16:44:38.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:44:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.979+0000 7fe935200640 1 Processor -- start 2024-09-13T16:44:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.979+0000 7fe935200640 1 -- start start 2024-09-13T16:44:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.980+0000 7fe935200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe930158b20 0x7fe9300c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.981+0000 7fe935200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93015c340 con 0x7fe9300c6a00 2024-09-13T16:44:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.981+0000 7fe935200640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe930158fd0 con 0x7fe930152f20 2024-09-13T16:44:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.981+0000 7fe935200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe930159b50 con 0x7fe930158b20 2024-09-13T16:44:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.981+0000 7fe92e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe9300c6a00 0x7fe9301634a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57160/0 (socket says 172.21.15.28:57160) 2024-09-13T16:44:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.981+0000 7fe92e800640 1 -- 172.21.15.28:0/4244224008 learned_addr learned my addr 172.21.15.28:0/4244224008 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.981+0000 7fe927e00640 1 --2- 172.21.15.28:0/4244224008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe930158b20 0x7fe9300c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.981+0000 7fe92d400640 1 -- 172.21.15.28:0/4244224008 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131516511 0 0) 0x7fe930158fd0 con 0x7fe930152f20 2024-09-13T16:44:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.981+0000 7fe927e00640 1 -- 172.21.15.28:0/4244224008 >> v1:172.21.15.191:6789/0 conn(0x7fe930152f20 legacy=0x7fe93015c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.981+0000 7fe927e00640 1 -- 172.21.15.28:0/4244224008 >> v1:172.21.15.28:6789/0 conn(0x7fe9300c6a00 legacy=0x7fe9301634a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.981+0000 7fe927e00640 1 -- 172.21.15.28:0/4244224008 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9301592b0 con 0x7fe930158b20 2024-09-13T16:44:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.982+0000 7fe927e00640 1 --2- 172.21.15.28:0/4244224008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe930158b20 0x7fe9300c5ec0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe9180097d0 tx=0x7fe91802a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a7099b3a3349b57 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.983+0000 7fe92d400640 1 -- 172.21.15.28:0/4244224008 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91802ae30 con 0x7fe930158b20 2024-09-13T16:44:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.983+0000 7fe92d400640 1 -- 172.21.15.28:0/4244224008 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe918032440 con 0x7fe930158b20 2024-09-13T16:44:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.983+0000 7fe92d400640 1 -- 172.21.15.28:0/4244224008 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9180327d0 con 0x7fe930158b20 2024-09-13T16:44:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.983+0000 7fe935200640 1 -- 172.21.15.28:0/4244224008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe930158b20 msgr2=0x7fe9300c5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.983+0000 7fe935200640 1 --2- 172.21.15.28:0/4244224008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe930158b20 0x7fe9300c5ec0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe9180097d0 tx=0x7fe91802a1a0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.983+0000 7fe935200640 1 -- 172.21.15.28:0/4244224008 shutdown_connections 2024-09-13T16:44:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.983+0000 7fe935200640 1 --2- 172.21.15.28:0/4244224008 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe930158b20 0x7fe9300c5ec0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.983+0000 7fe935200640 1 -- 172.21.15.28:0/4244224008 >> 172.21.15.28:0/4244224008 conn(0x7fe930066040 msgr2=0x7fe930069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.983+0000 7fe935200640 1 -- 172.21.15.28:0/4244224008 shutdown_connections 2024-09-13T16:44:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.983+0000 7fe935200640 1 -- 172.21.15.28:0/4244224008 wait complete. 2024-09-13T16:44:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.985+0000 7fe935200640 1 Processor -- start 2024-09-13T16:44:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.985+0000 7fe935200640 1 -- start start 2024-09-13T16:44:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.986+0000 7fe935200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9300c6a00 0x7fe9301f11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.986+0000 7fe935200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe930152f20 0x7fe93015b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.986+0000 7fe935200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe930158b20 0x7fe93015b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:38.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.986+0000 7fe935200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9301efcf0 con 0x7fe930158b20 2024-09-13T16:44:38.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.986+0000 7fe935200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9301f1cc0 con 0x7fe9300c6a00 2024-09-13T16:44:38.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.986+0000 7fe935200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe930146e40 con 0x7fe930152f20 2024-09-13T16:44:38.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.986+0000 7fe92e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe930158b20 0x7fe93015b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:38.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.986+0000 7fe92e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe930158b20 0x7fe93015b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60224/0 (socket says 172.21.15.28:60224) 2024-09-13T16:44:38.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.986+0000 7fe92e800640 1 -- 172.21.15.28:0/1082247950 learned_addr learned my addr 172.21.15.28:0/1082247950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:38.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.986+0000 7fe927e00640 1 --2- 172.21.15.28:0/1082247950 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe930152f20 0x7fe93015b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:38.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.986+0000 7fe92de00640 1 --2- 172.21.15.28:0/1082247950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9300c6a00 0x7fe9301f11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:38.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.987+0000 7fe92e800640 1 -- 172.21.15.28:0/1082247950 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe930152f20 msgr2=0x7fe93015b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.987+0000 7fe92e800640 1 --2- 172.21.15.28:0/1082247950 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe930152f20 0x7fe93015b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.987+0000 7fe92e800640 1 -- 172.21.15.28:0/1082247950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9300c6a00 msgr2=0x7fe9301f11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.987+0000 7fe92e800640 1 --2- 172.21.15.28:0/1082247950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9300c6a00 0x7fe9301f11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.987+0000 7fe92e800640 1 -- 172.21.15.28:0/1082247950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe918009400 con 0x7fe930158b20 2024-09-13T16:44:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.987+0000 7fe92e800640 1 --2- 172.21.15.28:0/1082247950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe930158b20 0x7fe93015b550 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe92800c8c0 tx=0x7fe92800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.987+0000 7fe926000640 1 -- 172.21.15.28:0/1082247950 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92800a640 con 0x7fe930158b20 2024-09-13T16:44:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.987+0000 7fe926000640 1 -- 172.21.15.28:0/1082247950 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92800a7a0 con 0x7fe930158b20 2024-09-13T16:44:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.987+0000 7fe926000640 1 -- 172.21.15.28:0/1082247950 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92800ab90 con 0x7fe930158b20 2024-09-13T16:44:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.987+0000 7fe935200640 1 -- 172.21.15.28:0/1082247950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9301f1f60 con 0x7fe930158b20 2024-09-13T16:44:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.988+0000 7fe935200640 1 -- 172.21.15.28:0/1082247950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9301f2520 con 0x7fe930158b20 2024-09-13T16:44:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.989+0000 7fe935200640 1 -- 172.21.15.28:0/1082247950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe930157640 con 0x7fe930158b20 2024-09-13T16:44:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.989+0000 7fe926000640 1 -- 172.21.15.28:0/1082247950 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe92800acf0 con 0x7fe930158b20 2024-09-13T16:44:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.990+0000 7fe926000640 1 --2- 172.21.15.28:0/1082247950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe90c06ac30 0x7fe90c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.990+0000 7fe926000640 1 -- 172.21.15.28:0/1082247950 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9280889e0 con 0x7fe930158b20 2024-09-13T16:44:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.991+0000 7fe92de00640 1 --2- 172.21.15.28:0/1082247950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe90c06ac30 0x7fe90c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.991+0000 7fe92de00640 1 --2- 172.21.15.28:0/1082247950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe90c06ac30 0x7fe90c06d0f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe920001120 tx=0x7fe920009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:38.994+0000 7fe926000640 1 -- 172.21.15.28:0/1082247950 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe928053bd0 con 0x7fe930158b20 2024-09-13T16:44:39.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.181+0000 7fe935200640 1 -- 172.21.15.28:0/1082247950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe93006a770 con 0x7fe930158b20 2024-09-13T16:44:39.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.183+0000 7fe926000640 1 -- 172.21.15.28:0/1082247950 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16481 (secure 0 0 0) 0x7fe928018070 con 0x7fe930158b20 2024-09-13T16:44:39.185 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:39.185 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:43:55.971358+0000","last_up_change":"2024-09-13T16:43:49.476313+0000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:43:49.643606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:43:52.828510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:43:53.945874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6801","nonce":2964323435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6803","nonce":2964323435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6807","nonce":2964323435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6805","nonce":2964323435}]},"public_addr":"172.21.15.28:6801/2964323435","cluster_addr":"172.21.15.28:6803/2964323435","heartbeat_back_addr":"172.21.15.28:6807/2964323435","heartbeat_front_addr":"172.21.15.28:6805/2964323435","state":["exists","up"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6809","nonce":23768570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6811","nonce":23768570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6815","nonce":23768570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6813","nonce":23768570}]},"public_addr":"172.21.15.28:6809/23768570","cluster_addr":"172.21.15.28:6811/23768570","heartbeat_back_addr":"172.21.15.28:6815/23768570","heartbeat_front_addr":"172.21.15.28:6813/23768570","state":["exists","up"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6817","nonce":886887303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6819","nonce":886887303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6823","nonce":886887303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6821","nonce":886887303}]},"public_addr":"172.21.15.28:6817/886887303","cluster_addr":"172.21.15.28:6819/886887303","heartbeat_back_addr":"172.21.15.28:6823/886887303","heartbeat_front_addr":"172.21.15.28:6821/886887303","state":["exists","up"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6825","nonce":2616036674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6827","nonce":2616036674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6831","nonce":2616036674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6829","nonce":2616036674}]},"public_addr":"172.21.15.28:6825/2616036674","cluster_addr":"172.21.15.28:6827/2616036674","heartbeat_back_addr":"172.21.15.28:6831/2616036674","heartbeat_front_addr":"172.21.15.28:6829/2616036674","state":["exists","up"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6801","nonce":2817102450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6803","nonce":2817102450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6807","nonce":2817102450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6805","nonce":2817102450}]},"public_addr":"172.21.15.191:6801/2817102450","cluster_addr":"172.21.15.191:6803/2817102450","heartbeat_back_addr":"172.21.15.191:6807/2817102450","heartbeat_front_addr":"172.21.15.191:6805/2817102450","state":["exists","up"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6817","nonce":2503027971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6819","nonce":2503027971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6823","nonce":2503027971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6821","nonce":2503027971}]},"public_addr":"172.21.15.191:6817/2503027971","cluster_addr":"172.21.15.191:6819/2503027971","heartbeat_back_addr":"172.21.15.191:6823/2503027971","heartbeat_front_addr":"172.21.15.191:6821/2503027971","state":["exists","up"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6809","nonce":3006595088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6811","nonce":3006595088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6815","nonce":3006595088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6813","nonce":3006595088}]},"public_addr":"172.21.15.191:6809/3006595088","cluster_addr":"172.21.15.191:6811/3006595088","heartbeat_back_addr":"172.21.15.191:6815/3006595088","heartbeat_front_addr":"172.21.15.191:6813/3006595088","state":["exists","up"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6825","nonce":1477327720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6827","nonce":1477327720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6831","nonce":1477327720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6829","nonce":1477327720}]},"public_addr":"172.21.15.191:6825/1477327720","cluster_addr":"172.21.15.191:6827/1477327720","heartbeat_back_addr":"172.21.15.191:6831/1477327720","heartbeat_front_addr":"172.21.15.191:6829/1477327720","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.917250+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.954351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.017329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.046221+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.707467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.726947+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.782321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.764850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:44:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.185+0000 7fe935200640 1 -- 172.21.15.28:0/1082247950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe90c06ac30 msgr2=0x7fe90c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.185+0000 7fe935200640 1 --2- 172.21.15.28:0/1082247950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe90c06ac30 0x7fe90c06d0f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe920001120 tx=0x7fe920009040 comp rx=0 tx=0).stop 2024-09-13T16:44:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.185+0000 7fe935200640 1 -- 172.21.15.28:0/1082247950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe930158b20 msgr2=0x7fe93015b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.185+0000 7fe935200640 1 --2- 172.21.15.28:0/1082247950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe930158b20 0x7fe93015b550 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe92800c8c0 tx=0x7fe92800cd80 comp rx=0 tx=0).stop 2024-09-13T16:44:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.185+0000 7fe935200640 1 -- 172.21.15.28:0/1082247950 shutdown_connections 2024-09-13T16:44:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.185+0000 7fe935200640 1 --2- 172.21.15.28:0/1082247950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe90c06ac30 0x7fe90c06d0f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.185+0000 7fe935200640 1 --2- 172.21.15.28:0/1082247950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe930158b20 0x7fe93015b550 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.185+0000 7fe935200640 1 --2- 172.21.15.28:0/1082247950 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe930152f20 0x7fe93015b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.185+0000 7fe935200640 1 --2- 172.21.15.28:0/1082247950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9300c6a00 0x7fe9301f11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.185+0000 7fe935200640 1 -- 172.21.15.28:0/1082247950 >> 172.21.15.28:0/1082247950 conn(0x7fe930066040 msgr2=0x7fe930156990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:39.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.186+0000 7fe935200640 1 -- 172.21.15.28:0/1082247950 shutdown_connections 2024-09-13T16:44:39.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.186+0000 7fe935200640 1 -- 172.21.15.28:0/1082247950 wait complete. 2024-09-13T16:44:39.205 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:43:49.643606+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:43:52.828510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:43:53.945874+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:44:39.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:44:39.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.362+0000 7f7563600640 1 Processor -- start 2024-09-13T16:44:39.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.362+0000 7f7563600640 1 -- start start 2024-09-13T16:44:39.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.363+0000 7f7563600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f755c14f9a0 0x7f755c0c5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:39.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.363+0000 7f7563600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f755c14fd80 0x7f755c0c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:39.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.363+0000 7f7563600640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f755c0c44e0 0x7f755c0c9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:39.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.363+0000 7f7563600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f755c0c2580 con 0x7f755c14f9a0 2024-09-13T16:44:39.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.363+0000 7f7563600640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f755c155800 con 0x7f755c0c44e0 2024-09-13T16:44:39.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.363+0000 7f7563600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755c1567d0 con 0x7f755c14fd80 2024-09-13T16:44:39.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.364+0000 7f7560c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f755c14f9a0 0x7f755c0c5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:39.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.364+0000 7f7560c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f755c14f9a0 0x7f755c0c5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60232/0 (socket says 172.21.15.28:60232) 2024-09-13T16:44:39.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.364+0000 7f7560c00640 1 -- 172.21.15.28:0/2618098793 learned_addr learned my addr 172.21.15.28:0/2618098793 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:39.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.364+0000 7f755be00640 1 --2- 172.21.15.28:0/2618098793 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f755c14fd80 0x7f755c0c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:39.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.364+0000 7f7561600640 1 --2- 172.21.15.28:0/2618098793 >> v2:172.21.15.191:3300/0 conn(0x7f755c0c44e0 0x7f755c0c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:39.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.364+0000 7f7560c00640 1 -- 172.21.15.28:0/2618098793 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f755c14fd80 msgr2=0x7f755c0c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:39.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.364+0000 7f7560c00640 1 --2- 172.21.15.28:0/2618098793 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f755c14fd80 0x7f755c0c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.364+0000 7f7560c00640 1 -- 172.21.15.28:0/2618098793 >> v2:172.21.15.191:3300/0 conn(0x7f755c0c44e0 msgr2=0x7f755c0c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:39.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.364+0000 7f7560c00640 1 --2- 172.21.15.28:0/2618098793 >> v2:172.21.15.191:3300/0 conn(0x7f755c0c44e0 0x7f755c0c9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.364+0000 7f7560c00640 1 -- 172.21.15.28:0/2618098793 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c0c1ef0 con 0x7f755c14f9a0 2024-09-13T16:44:39.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.365+0000 7f7560c00640 1 --2- 172.21.15.28:0/2618098793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f755c14f9a0 0x7f755c0c5570 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f755c072630 tx=0x7f755002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fca39960becb935 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:39.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.365+0000 7f755b400640 1 -- 172.21.15.28:0/2618098793 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7550032440 con 0x7f755c14f9a0 2024-09-13T16:44:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.365+0000 7f755b400640 1 -- 172.21.15.28:0/2618098793 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75500325a0 con 0x7f755c14f9a0 2024-09-13T16:44:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.366+0000 7f755b400640 1 -- 172.21.15.28:0/2618098793 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7550032930 con 0x7f755c14f9a0 2024-09-13T16:44:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.366+0000 7f7563600640 1 -- 172.21.15.28:0/2618098793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f755c14f9a0 msgr2=0x7f755c0c5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.366+0000 7f7563600640 1 --2- 172.21.15.28:0/2618098793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f755c14f9a0 0x7f755c0c5570 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f755c072630 tx=0x7f755002a270 comp rx=0 tx=0).stop 2024-09-13T16:44:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.366+0000 7f7563600640 1 -- 172.21.15.28:0/2618098793 shutdown_connections 2024-09-13T16:44:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.366+0000 7f7563600640 1 --2- 172.21.15.28:0/2618098793 >> v2:172.21.15.191:3300/0 conn(0x7f755c0c44e0 0x7f755c0c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.366+0000 7f7563600640 1 --2- 172.21.15.28:0/2618098793 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f755c14fd80 0x7f755c0c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.366+0000 7f7563600640 1 --2- 172.21.15.28:0/2618098793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f755c14f9a0 0x7f755c0c5570 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.366+0000 7f7563600640 1 -- 172.21.15.28:0/2618098793 >> 172.21.15.28:0/2618098793 conn(0x7f755c0ca2f0 msgr2=0x7f755c0ca6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.366+0000 7f7563600640 1 -- 172.21.15.28:0/2618098793 shutdown_connections 2024-09-13T16:44:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.366+0000 7f7563600640 1 -- 172.21.15.28:0/2618098793 wait complete. 2024-09-13T16:44:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.368+0000 7f7563600640 1 Processor -- start 2024-09-13T16:44:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.368+0000 7f7563600640 1 -- start start 2024-09-13T16:44:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.369+0000 7f7563600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f755c0c44e0 0x7f755c06d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.369+0000 7f7563600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f755c14f9a0 0x7f755c1570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.369+0000 7f7563600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f755c151cb0 0x7f755c158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.369+0000 7f7563600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c157df0 con 0x7f755c151cb0 2024-09-13T16:44:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.369+0000 7f7563600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c152a80 con 0x7f755c0c44e0 2024-09-13T16:44:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.369+0000 7f7563600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755c152d40 con 0x7f755c14f9a0 2024-09-13T16:44:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.369+0000 7f7561600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f755c151cb0 0x7f755c158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.369+0000 7f7561600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f755c151cb0 0x7f755c158560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60238/0 (socket says 172.21.15.28:60238) 2024-09-13T16:44:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.369+0000 7f7561600640 1 -- 172.21.15.28:0/1479858419 learned_addr learned my addr 172.21.15.28:0/1479858419 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.369+0000 7f755be00640 1 --2- 172.21.15.28:0/1479858419 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f755c14f9a0 0x7f755c1570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.369+0000 7f7560c00640 1 --2- 172.21.15.28:0/1479858419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f755c0c44e0 0x7f755c06d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.370+0000 7f7561600640 1 -- 172.21.15.28:0/1479858419 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f755c14f9a0 msgr2=0x7f755c1570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.370+0000 7f7561600640 1 --2- 172.21.15.28:0/1479858419 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f755c14f9a0 0x7f755c1570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.370+0000 7f7561600640 1 -- 172.21.15.28:0/1479858419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f755c0c44e0 msgr2=0x7f755c06d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.370+0000 7f7561600640 1 --2- 172.21.15.28:0/1479858419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f755c0c44e0 0x7f755c06d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.370+0000 7f7561600640 1 -- 172.21.15.28:0/1479858419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75500094d0 con 0x7f755c151cb0 2024-09-13T16:44:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.370+0000 7f7561600640 1 --2- 172.21.15.28:0/1479858419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f755c151cb0 0x7f755c158560 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f755400eed0 tx=0x7f755400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.370+0000 7f7558c00640 1 -- 172.21.15.28:0/1479858419 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7554010040 con 0x7f755c151cb0 2024-09-13T16:44:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.370+0000 7f7558c00640 1 -- 172.21.15.28:0/1479858419 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7554014400 con 0x7f755c151cb0 2024-09-13T16:44:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.370+0000 7f7558c00640 1 -- 172.21.15.28:0/1479858419 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75540147b0 con 0x7f755c151cb0 2024-09-13T16:44:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.370+0000 7f7563600640 1 -- 172.21.15.28:0/1479858419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f755c153020 con 0x7f755c151cb0 2024-09-13T16:44:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.371+0000 7f7563600640 1 -- 172.21.15.28:0/1479858419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f755c152260 con 0x7f755c151cb0 2024-09-13T16:44:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.371+0000 7f7563600640 1 -- 172.21.15.28:0/1479858419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755c153210 con 0x7f755c151cb0 2024-09-13T16:44:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.372+0000 7f7558c00640 1 -- 172.21.15.28:0/1479858419 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7554014910 con 0x7f755c151cb0 2024-09-13T16:44:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.373+0000 7f7558c00640 1 --2- 172.21.15.28:0/1479858419 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f753c06ac30 0x7f753c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.373+0000 7f7558c00640 1 -- 172.21.15.28:0/1479858419 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7554089d80 con 0x7f755c151cb0 2024-09-13T16:44:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.374+0000 7f7560c00640 1 --2- 172.21.15.28:0/1479858419 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f753c06ac30 0x7f753c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.374+0000 7f7560c00640 1 --2- 172.21.15.28:0/1479858419 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f753c06ac30 0x7f753c06d0f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7550034000 tx=0x7f755002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.377+0000 7f7558c00640 1 -- 172.21.15.28:0/1479858419 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f755c153210 con 0x7f755c151cb0 2024-09-13T16:44:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.558+0000 7f7563600640 1 -- 172.21.15.28:0/1479858419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f755c153210 con 0x7f755c151cb0 2024-09-13T16:44:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.559+0000 7f7558c00640 1 -- 172.21.15.28:0/1479858419 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7554018020 con 0x7f755c151cb0 2024-09-13T16:44:39.560 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T16:44:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.561+0000 7f7563600640 1 -- 172.21.15.28:0/1479858419 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f753c06ac30 msgr2=0x7f753c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.561+0000 7f7563600640 1 --2- 172.21.15.28:0/1479858419 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f753c06ac30 0x7f753c06d0f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7550034000 tx=0x7f755002e000 comp rx=0 tx=0).stop 2024-09-13T16:44:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.561+0000 7f7563600640 1 -- 172.21.15.28:0/1479858419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f755c151cb0 msgr2=0x7f755c158560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.561+0000 7f7563600640 1 --2- 172.21.15.28:0/1479858419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f755c151cb0 0x7f755c158560 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f755400eed0 tx=0x7f755400b480 comp rx=0 tx=0).stop 2024-09-13T16:44:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.561+0000 7f7563600640 1 -- 172.21.15.28:0/1479858419 shutdown_connections 2024-09-13T16:44:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.561+0000 7f7563600640 1 --2- 172.21.15.28:0/1479858419 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f753c06ac30 0x7f753c06d0f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.561+0000 7f7563600640 1 --2- 172.21.15.28:0/1479858419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f755c151cb0 0x7f755c158560 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.561+0000 7f7563600640 1 --2- 172.21.15.28:0/1479858419 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f755c14f9a0 0x7f755c1570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.561+0000 7f7563600640 1 --2- 172.21.15.28:0/1479858419 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f755c0c44e0 0x7f755c06d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.561+0000 7f7563600640 1 -- 172.21.15.28:0/1479858419 >> 172.21.15.28:0/1479858419 conn(0x7f755c0ca2f0 msgr2=0x7f755c153460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.561+0000 7f7563600640 1 -- 172.21.15.28:0/1479858419 shutdown_connections 2024-09-13T16:44:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.561+0000 7f7563600640 1 -- 172.21.15.28:0/1479858419 wait complete. 2024-09-13T16:44:39.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:44:39.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.737+0000 7f396b200640 1 Processor -- start 2024-09-13T16:44:39.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.738+0000 7f396b200640 1 -- start start 2024-09-13T16:44:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.739+0000 7f396b200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39640c7f10 0x7f39640c6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.739+0000 7f396b200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f39640c82f0 0x7f3964155790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.739+0000 7f396b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396406f760 con 0x7f39640c5c00 2024-09-13T16:44:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.739+0000 7f396b200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39640c6e70 con 0x7f39640c82f0 2024-09-13T16:44:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.739+0000 7f396b200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39640c7110 con 0x7f39640c7f10 2024-09-13T16:44:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.739+0000 7f3969400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f39640c5c00 0x7f3964158bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57176/0 (socket says 172.21.15.28:57176) 2024-09-13T16:44:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.739+0000 7f3969400640 1 -- 172.21.15.28:0/4277956016 learned_addr learned my addr 172.21.15.28:0/4277956016 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.739+0000 7f3969e00640 1 --2- 172.21.15.28:0/4277956016 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39640c7f10 0x7f39640c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.739+0000 7f396a800640 1 --2- 172.21.15.28:0/4277956016 >> v2:172.21.15.191:3300/0 conn(0x7f39640c82f0 0x7f3964155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:39.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.739+0000 7f3968a00640 1 -- 172.21.15.28:0/4277956016 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803350355 0 0) 0x7f396406f760 con 0x7f39640c5c00 2024-09-13T16:44:39.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.739+0000 7f3968a00640 1 -- 172.21.15.28:0/4277956016 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f394c003520 con 0x7f39640c5c00 2024-09-13T16:44:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.740+0000 7f3969e00640 1 -- 172.21.15.28:0/4277956016 >> v2:172.21.15.191:3300/0 conn(0x7f39640c82f0 msgr2=0x7f3964155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.740+0000 7f3969e00640 1 --2- 172.21.15.28:0/4277956016 >> v2:172.21.15.191:3300/0 conn(0x7f39640c82f0 0x7f3964155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.740+0000 7f3969e00640 1 -- 172.21.15.28:0/4277956016 >> v1:172.21.15.28:6789/0 conn(0x7f39640c5c00 legacy=0x7f3964158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.740+0000 7f3969e00640 1 -- 172.21.15.28:0/4277956016 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396415c120 con 0x7f39640c7f10 2024-09-13T16:44:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.740+0000 7f396a800640 1 --2- 172.21.15.28:0/4277956016 >> v2:172.21.15.191:3300/0 conn(0x7f39640c82f0 0x7f3964155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.741+0000 7f3969e00640 1 --2- 172.21.15.28:0/4277956016 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39640c7f10 0x7f39640c6610 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f39580097a0 tx=0x7f395802a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6d91702f189f89d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.741+0000 7f3968a00640 1 -- 172.21.15.28:0/4277956016 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958032440 con 0x7f39640c7f10 2024-09-13T16:44:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.741+0000 7f3968a00640 1 -- 172.21.15.28:0/4277956016 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39580325a0 con 0x7f39640c7f10 2024-09-13T16:44:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.741+0000 7f3968a00640 1 -- 172.21.15.28:0/4277956016 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958032930 con 0x7f39640c7f10 2024-09-13T16:44:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.741+0000 7f396b200640 1 -- 172.21.15.28:0/4277956016 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39640c7f10 msgr2=0x7f39640c6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.741+0000 7f396b200640 1 --2- 172.21.15.28:0/4277956016 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39640c7f10 0x7f39640c6610 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f39580097a0 tx=0x7f395802a230 comp rx=0 tx=0).stop 2024-09-13T16:44:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.742+0000 7f396b200640 1 -- 172.21.15.28:0/4277956016 shutdown_connections 2024-09-13T16:44:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.742+0000 7f396b200640 1 --2- 172.21.15.28:0/4277956016 >> v2:172.21.15.191:3300/0 conn(0x7f39640c82f0 0x7f3964155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.742+0000 7f396b200640 1 --2- 172.21.15.28:0/4277956016 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39640c7f10 0x7f39640c6610 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.742+0000 7f396b200640 1 -- 172.21.15.28:0/4277956016 >> 172.21.15.28:0/4277956016 conn(0x7f396406af90 msgr2=0x7f396406b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.742+0000 7f396b200640 1 -- 172.21.15.28:0/4277956016 shutdown_connections 2024-09-13T16:44:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.742+0000 7f396b200640 1 -- 172.21.15.28:0/4277956016 wait complete. 2024-09-13T16:44:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.743+0000 7f396b200640 1 Processor -- start 2024-09-13T16:44:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.743+0000 7f396b200640 1 -- start start 2024-09-13T16:44:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.744+0000 7f396b200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39640c5c00 0x7f39641f1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.744+0000 7f396b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39640c7f10 0x7f3964155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.744+0000 7f396b200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39640c82f0 0x7f39641ee5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.744+0000 7f396b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39640bb9a0 con 0x7f39640c7f10 2024-09-13T16:44:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.744+0000 7f396b200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39641f2460 con 0x7f39640c82f0 2024-09-13T16:44:39.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.744+0000 7f396b200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39640bb6c0 con 0x7f39640c5c00 2024-09-13T16:44:39.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.745+0000 7f3969400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39640c7f10 0x7f3964155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:39.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.745+0000 7f3969400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39640c7f10 0x7f3964155550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60248/0 (socket says 172.21.15.28:60248) 2024-09-13T16:44:39.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.745+0000 7f3969400640 1 -- 172.21.15.28:0/3160625389 learned_addr learned my addr 172.21.15.28:0/3160625389 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:39.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.745+0000 7f3969e00640 1 --2- 172.21.15.28:0/3160625389 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39640c5c00 0x7f39641f1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:39.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.745+0000 7f396a800640 1 --2- 172.21.15.28:0/3160625389 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39640c82f0 0x7f39641ee5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:39.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.745+0000 7f3969400640 1 -- 172.21.15.28:0/3160625389 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39640c5c00 msgr2=0x7f39641f1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:39.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.745+0000 7f3969400640 1 --2- 172.21.15.28:0/3160625389 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39640c5c00 0x7f39641f1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.745+0000 7f3969400640 1 -- 172.21.15.28:0/3160625389 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39640c82f0 msgr2=0x7f39641ee5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:39.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.745+0000 7f3969400640 1 --2- 172.21.15.28:0/3160625389 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39640c82f0 0x7f39641ee5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.745+0000 7f3969400640 1 -- 172.21.15.28:0/3160625389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3960009a70 con 0x7f39640c7f10 2024-09-13T16:44:39.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.745+0000 7f3969400640 1 --2- 172.21.15.28:0/3160625389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39640c7f10 0x7f3964155550 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f395400c500 tx=0x7f395400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:39.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.745+0000 7f395e000640 1 -- 172.21.15.28:0/3160625389 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3954005ce0 con 0x7f39640c7f10 2024-09-13T16:44:39.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.745+0000 7f395e000640 1 -- 172.21.15.28:0/3160625389 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3954005e40 con 0x7f39640c7f10 2024-09-13T16:44:39.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.745+0000 7f395e000640 1 -- 172.21.15.28:0/3160625389 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3954007310 con 0x7f39640c7f10 2024-09-13T16:44:39.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.746+0000 7f396b200640 1 -- 172.21.15.28:0/3160625389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3958009400 con 0x7f39640c7f10 2024-09-13T16:44:39.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.746+0000 7f396b200640 1 -- 172.21.15.28:0/3160625389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3964159770 con 0x7f39640c7f10 2024-09-13T16:44:39.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.747+0000 7f396b200640 1 -- 172.21.15.28:0/3160625389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39640c6610 con 0x7f39640c7f10 2024-09-13T16:44:39.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.748+0000 7f395e000640 1 -- 172.21.15.28:0/3160625389 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3954007530 con 0x7f39640c7f10 2024-09-13T16:44:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.748+0000 7f395e000640 1 --2- 172.21.15.28:0/3160625389 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f394806ac00 0x7f394806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.749+0000 7f395e000640 1 -- 172.21.15.28:0/3160625389 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3954088920 con 0x7f39640c7f10 2024-09-13T16:44:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.749+0000 7f3969e00640 1 --2- 172.21.15.28:0/3160625389 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f394806ac00 0x7f394806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.749+0000 7f3969e00640 1 --2- 172.21.15.28:0/3160625389 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f394806ac00 0x7f394806d0c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3958034000 tx=0x7f395802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.753+0000 7f395e000640 1 -- 172.21.15.28:0/3160625389 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3954053c60 con 0x7f39640c7f10 2024-09-13T16:44:39.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.932+0000 7f396b200640 1 -- 172.21.15.28:0/3160625389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f39640c9580 con 0x7f39640c7f10 2024-09-13T16:44:39.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.933+0000 7f395e000640 1 -- 172.21.15.28:0/3160625389 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3954017020 con 0x7f39640c7f10 2024-09-13T16:44:39.935 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T16:44:39.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.935+0000 7f396b200640 1 -- 172.21.15.28:0/3160625389 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f394806ac00 msgr2=0x7f394806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:39.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.935+0000 7f396b200640 1 --2- 172.21.15.28:0/3160625389 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f394806ac00 0x7f394806d0c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3958034000 tx=0x7f395802e000 comp rx=0 tx=0).stop 2024-09-13T16:44:39.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.935+0000 7f396b200640 1 -- 172.21.15.28:0/3160625389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39640c7f10 msgr2=0x7f3964155550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:39.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.935+0000 7f396b200640 1 --2- 172.21.15.28:0/3160625389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39640c7f10 0x7f3964155550 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f395400c500 tx=0x7f395400c900 comp rx=0 tx=0).stop 2024-09-13T16:44:39.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.935+0000 7f396b200640 1 -- 172.21.15.28:0/3160625389 shutdown_connections 2024-09-13T16:44:39.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.935+0000 7f396b200640 1 --2- 172.21.15.28:0/3160625389 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f394806ac00 0x7f394806d0c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.935+0000 7f396b200640 1 --2- 172.21.15.28:0/3160625389 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39640c82f0 0x7f39641ee5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.935+0000 7f396b200640 1 --2- 172.21.15.28:0/3160625389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39640c7f10 0x7f3964155550 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.935+0000 7f396b200640 1 --2- 172.21.15.28:0/3160625389 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39640c5c00 0x7f39641f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:39.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.935+0000 7f396b200640 1 -- 172.21.15.28:0/3160625389 >> 172.21.15.28:0/3160625389 conn(0x7f396406af90 msgr2=0x7f3964070db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:39.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.935+0000 7f396b200640 1 -- 172.21.15.28:0/3160625389 shutdown_connections 2024-09-13T16:44:39.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:39.935+0000 7f396b200640 1 -- 172.21.15.28:0/3160625389 wait complete. 2024-09-13T16:44:39.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:44:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.110+0000 7f0d98c00640 1 Processor -- start 2024-09-13T16:44:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.110+0000 7f0d98c00640 1 -- start start 2024-09-13T16:44:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.111+0000 7f0d98c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0d9415a440 0x7f0d940c5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.111+0000 7f0d98c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d940c1d80 con 0x7f0d9406dcb0 2024-09-13T16:44:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.111+0000 7f0d98c00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d9415a920 con 0x7f0d9415a060 2024-09-13T16:44:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.111+0000 7f0d98c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d9415ac00 con 0x7f0d9415a440 2024-09-13T16:44:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.111+0000 7f0d8be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0d9415a440 0x7f0d940c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.111+0000 7f0d92800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0d9406dcb0 0x7f0d941504e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57188/0 (socket says 172.21.15.28:57188) 2024-09-13T16:44:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.111+0000 7f0d92800640 1 -- 172.21.15.28:0/2379267130 learned_addr learned my addr 172.21.15.28:0/2379267130 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.112+0000 7f0d8be00640 1 -- 172.21.15.28:0/2379267130 >> v1:172.21.15.191:6789/0 conn(0x7f0d9415a060 legacy=0x7f0d9406d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.112+0000 7f0d8be00640 1 -- 172.21.15.28:0/2379267130 >> v1:172.21.15.28:6789/0 conn(0x7f0d9406dcb0 legacy=0x7f0d941504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.112+0000 7f0d8be00640 1 -- 172.21.15.28:0/2379267130 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d940c2d50 con 0x7f0d9415a440 2024-09-13T16:44:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.112+0000 7f0d91400640 1 -- 172.21.15.28:0/2379267130 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428333150 0 0) 0x7f0d940c1d80 con 0x7f0d9406dcb0 2024-09-13T16:44:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.113+0000 7f0d8be00640 1 --2- 172.21.15.28:0/2379267130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9415a440 0x7f0d940c5500 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0097a0 tx=0x7f0d7c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ea75b4e201a01b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.113+0000 7f0d91400640 1 -- 172.21.15.28:0/2379267130 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c032440 con 0x7f0d9415a440 2024-09-13T16:44:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.113+0000 7f0d91400640 1 -- 172.21.15.28:0/2379267130 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7c0325a0 con 0x7f0d9415a440 2024-09-13T16:44:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.113+0000 7f0d91400640 1 -- 172.21.15.28:0/2379267130 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c032930 con 0x7f0d9415a440 2024-09-13T16:44:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.114+0000 7f0d98c00640 1 -- 172.21.15.28:0/2379267130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9415a440 msgr2=0x7f0d940c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.114+0000 7f0d98c00640 1 --2- 172.21.15.28:0/2379267130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9415a440 0x7f0d940c5500 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0097a0 tx=0x7f0d7c02a230 comp rx=0 tx=0).stop 2024-09-13T16:44:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.114+0000 7f0d98c00640 1 -- 172.21.15.28:0/2379267130 shutdown_connections 2024-09-13T16:44:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.114+0000 7f0d98c00640 1 --2- 172.21.15.28:0/2379267130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9415a440 0x7f0d940c5500 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.114+0000 7f0d98c00640 1 -- 172.21.15.28:0/2379267130 >> 172.21.15.28:0/2379267130 conn(0x7f0d940ca6b0 msgr2=0x7f0d940caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.114+0000 7f0d98c00640 1 -- 172.21.15.28:0/2379267130 shutdown_connections 2024-09-13T16:44:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.114+0000 7f0d98c00640 1 -- 172.21.15.28:0/2379267130 wait complete. 2024-09-13T16:44:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.115+0000 7f0d98c00640 1 Processor -- start 2024-09-13T16:44:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.115+0000 7f0d98c00640 1 -- start start 2024-09-13T16:44:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.116+0000 7f0d98c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9406dcb0 0x7f0d940c6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.116+0000 7f0d98c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d9415a060 0x7f0d941ef070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:40.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.116+0000 7f0d98c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d9415a440 0x7f0d941ef5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:40.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.116+0000 7f0d98c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d941eebe0 con 0x7f0d9406dcb0 2024-09-13T16:44:40.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.116+0000 7f0d98c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d940c7c50 con 0x7f0d9415a060 2024-09-13T16:44:40.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.116+0000 7f0d98c00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d940c70d0 con 0x7f0d9415a440 2024-09-13T16:44:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.117+0000 7f0d91e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9406dcb0 0x7f0d940c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.117+0000 7f0d91e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9406dcb0 0x7f0d940c6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60268/0 (socket says 172.21.15.28:60268) 2024-09-13T16:44:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.117+0000 7f0d91e00640 1 -- 172.21.15.28:0/182291392 learned_addr learned my addr 172.21.15.28:0/182291392 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.117+0000 7f0d8be00640 1 --2- 172.21.15.28:0/182291392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d9415a060 0x7f0d941ef070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.117+0000 7f0d92800640 1 --2- 172.21.15.28:0/182291392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d9415a440 0x7f0d941ef5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.117+0000 7f0d91e00640 1 -- 172.21.15.28:0/182291392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d9415a440 msgr2=0x7f0d941ef5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.117+0000 7f0d91e00640 1 --2- 172.21.15.28:0/182291392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d9415a440 0x7f0d941ef5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.117+0000 7f0d91e00640 1 -- 172.21.15.28:0/182291392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d9415a060 msgr2=0x7f0d941ef070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.117+0000 7f0d91e00640 1 --2- 172.21.15.28:0/182291392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d9415a060 0x7f0d941ef070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.117+0000 7f0d91e00640 1 -- 172.21.15.28:0/182291392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c009400 con 0x7f0d9406dcb0 2024-09-13T16:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.117+0000 7f0d91e00640 1 --2- 172.21.15.28:0/182291392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9406dcb0 0x7f0d940c6720 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0d8400c450 tx=0x7f0d8400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.118+0000 7f0d8a000640 1 -- 172.21.15.28:0/182291392 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d840052d0 con 0x7f0d9406dcb0 2024-09-13T16:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.118+0000 7f0d8a000640 1 -- 172.21.15.28:0/182291392 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d84005430 con 0x7f0d9406dcb0 2024-09-13T16:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.118+0000 7f0d98c00640 1 -- 172.21.15.28:0/182291392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d940c73b0 con 0x7f0d9406dcb0 2024-09-13T16:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.118+0000 7f0d8a000640 1 -- 172.21.15.28:0/182291392 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d84012870 con 0x7f0d9406dcb0 2024-09-13T16:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.118+0000 7f0d98c00640 1 -- 172.21.15.28:0/182291392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d940c7970 con 0x7f0d9406dcb0 2024-09-13T16:44:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.119+0000 7f0d98c00640 1 -- 172.21.15.28:0/182291392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d941ee9d0 con 0x7f0d9406dcb0 2024-09-13T16:44:40.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.120+0000 7f0d8a000640 1 -- 172.21.15.28:0/182291392 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0d840129d0 con 0x7f0d9406dcb0 2024-09-13T16:44:40.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.120+0000 7f0d8a000640 1 --2- 172.21.15.28:0/182291392 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0d7006ac30 0x7f0d7006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:40.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.121+0000 7f0d8a000640 1 -- 172.21.15.28:0/182291392 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d84088ea0 con 0x7f0d9406dcb0 2024-09-13T16:44:40.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.121+0000 7f0d8be00640 1 --2- 172.21.15.28:0/182291392 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0d7006ac30 0x7f0d7006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:40.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.121+0000 7f0d8be00640 1 --2- 172.21.15.28:0/182291392 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0d7006ac30 0x7f0d7006d0f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c034000 tx=0x7f0d7c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:40.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.125+0000 7f0d8a000640 1 -- 172.21.15.28:0/182291392 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0d84054000 con 0x7f0d9406dcb0 2024-09-13T16:44:40.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.302+0000 7f0d98c00640 1 -- 172.21.15.28:0/182291392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0d940c62a0 con 0x7f0d9406dcb0 2024-09-13T16:44:40.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.303+0000 7f0d8a000640 1 -- 172.21.15.28:0/182291392 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0d84057620 con 0x7f0d9406dcb0 2024-09-13T16:44:40.304 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T16:44:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.305+0000 7f0d98c00640 1 -- 172.21.15.28:0/182291392 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0d7006ac30 msgr2=0x7f0d7006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.305+0000 7f0d98c00640 1 --2- 172.21.15.28:0/182291392 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0d7006ac30 0x7f0d7006d0f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c034000 tx=0x7f0d7c02e000 comp rx=0 tx=0).stop 2024-09-13T16:44:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.305+0000 7f0d98c00640 1 -- 172.21.15.28:0/182291392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9406dcb0 msgr2=0x7f0d940c6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.305+0000 7f0d98c00640 1 --2- 172.21.15.28:0/182291392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9406dcb0 0x7f0d940c6720 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0d8400c450 tx=0x7f0d8400c910 comp rx=0 tx=0).stop 2024-09-13T16:44:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.305+0000 7f0d98c00640 1 -- 172.21.15.28:0/182291392 shutdown_connections 2024-09-13T16:44:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.305+0000 7f0d98c00640 1 --2- 172.21.15.28:0/182291392 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0d7006ac30 0x7f0d7006d0f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.305+0000 7f0d98c00640 1 --2- 172.21.15.28:0/182291392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d9415a440 0x7f0d941ef5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.305+0000 7f0d98c00640 1 --2- 172.21.15.28:0/182291392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d9415a060 0x7f0d941ef070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.305+0000 7f0d98c00640 1 --2- 172.21.15.28:0/182291392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9406dcb0 0x7f0d940c6720 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.305+0000 7f0d98c00640 1 -- 172.21.15.28:0/182291392 >> 172.21.15.28:0/182291392 conn(0x7f0d940ca6b0 msgr2=0x7f0d9414adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:40.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.305+0000 7f0d98c00640 1 -- 172.21.15.28:0/182291392 shutdown_connections 2024-09-13T16:44:40.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.305+0000 7f0d98c00640 1 -- 172.21.15.28:0/182291392 wait complete. 2024-09-13T16:44:40.321 DEBUG:tasks.mon_thrash:Yielding 2024-09-13T16:44:40.321 INFO:teuthology.run_tasks:Running task workunit... 2024-09-13T16:44:40.481 INFO:tasks.workunit:Pulling workunits from ref 50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T16:44:40.482 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-13T16:44:40.482 INFO:tasks.workunit:timeout=3h 2024-09-13T16:44:40.482 INFO:tasks.workunit:cleanup=True 2024-09-13T16:44:40.482 DEBUG:teuthology.orchestra.run.smithi028:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:44:40.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:44:40.522 INFO:teuthology.orchestra.run.smithi028.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-13T16:44:40.522 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-09-13T16:44:40.522 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 29h/41d Inode: 1 Links: 2 2024-09-13T16:44:40.522 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T16:44:40.522 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-09-13T16:44:40.523 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-09-13T16:44:40.523 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-13 16:43:59.973045605 +0000 2024-09-13T16:44:40.523 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-13 16:44:36.090930084 +0000 2024-09-13T16:44:40.523 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-13 16:43:59.973045605 +0000 2024-09-13T16:44:40.523 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-13T16:44:40.523 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-13T16:44:40.627 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T16:44:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.643+0000 7f5c13e00640 1 Processor -- start 2024-09-13T16:44:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.643+0000 7f5c13e00640 1 -- start start 2024-09-13T16:44:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.644+0000 7f5c13e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5c141509b0 0x7f5c1406cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.644+0000 7f5c13e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c140c1b40 0x7f5c1406d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.644+0000 7f5c13e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c1406e100 con 0x7f5c140c6240 2024-09-13T16:44:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.644+0000 7f5c13e00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c14150d90 con 0x7f5c141509b0 2024-09-13T16:44:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.644+0000 7f5c13e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c140c43b0 con 0x7f5c140c1b40 2024-09-13T16:44:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.645+0000 7f5c13400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5c140c6240 0x7f5c14156990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57190/0 (socket says 172.21.15.28:57190) 2024-09-13T16:44:40.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.645+0000 7f5c13400640 1 -- 172.21.15.28:0/342719926 learned_addr learned my addr 172.21.15.28:0/342719926 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:40.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.645+0000 7f5c12a00640 1 --2- 172.21.15.28:0/342719926 >> v2:172.21.15.191:3300/0 conn(0x7f5c141509b0 0x7f5c1406cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:40.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.645+0000 7f5c12000640 1 --2- 172.21.15.28:0/342719926 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c140c1b40 0x7f5c1406d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.645+0000 7f5c12a00640 1 -- 172.21.15.28:0/342719926 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c140c1b40 msgr2=0x7f5c1406d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.645+0000 7f5c11600640 1 -- 172.21.15.28:0/342719926 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999522639 0 0) 0x7f5c1406e100 con 0x7f5c140c6240 2024-09-13T16:44:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.645+0000 7f5c12a00640 1 --2- 172.21.15.28:0/342719926 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c140c1b40 0x7f5c1406d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.645+0000 7f5c12a00640 1 -- 172.21.15.28:0/342719926 >> v1:172.21.15.28:6789/0 conn(0x7f5c140c6240 legacy=0x7f5c14156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.645+0000 7f5c12a00640 1 -- 172.21.15.28:0/342719926 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c1406d7d0 con 0x7f5c141509b0 2024-09-13T16:44:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.645+0000 7f5c12000640 1 --2- 172.21.15.28:0/342719926 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c140c1b40 0x7f5c1406d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.647+0000 7f5c12a00640 1 --2- 172.21.15.28:0/342719926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c141509b0 0x7f5c1406cba0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5c040097a0 tx=0x7f5c0402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e5f6ef53d5e24af server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:40.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.647+0000 7f5c11600640 1 -- 172.21.15.28:0/342719926 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c04032440 con 0x7f5c141509b0 2024-09-13T16:44:40.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.647+0000 7f5c11600640 1 -- 172.21.15.28:0/342719926 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c040325a0 con 0x7f5c141509b0 2024-09-13T16:44:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.647+0000 7f5c11600640 1 -- 172.21.15.28:0/342719926 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c04032930 con 0x7f5c141509b0 2024-09-13T16:44:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.647+0000 7f5c13e00640 1 -- 172.21.15.28:0/342719926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c141509b0 msgr2=0x7f5c1406cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.647+0000 7f5c13e00640 1 --2- 172.21.15.28:0/342719926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c141509b0 0x7f5c1406cba0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5c040097a0 tx=0x7f5c0402a180 comp rx=0 tx=0).stop 2024-09-13T16:44:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.648+0000 7f5c13e00640 1 -- 172.21.15.28:0/342719926 shutdown_connections 2024-09-13T16:44:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.648+0000 7f5c13e00640 1 --2- 172.21.15.28:0/342719926 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c140c1b40 0x7f5c1406d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.648+0000 7f5c13e00640 1 --2- 172.21.15.28:0/342719926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c141509b0 0x7f5c1406cba0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.648+0000 7f5c13e00640 1 -- 172.21.15.28:0/342719926 >> 172.21.15.28:0/342719926 conn(0x7f5c140ca0e0 msgr2=0x7f5c140ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.648+0000 7f5c13e00640 1 -- 172.21.15.28:0/342719926 shutdown_connections 2024-09-13T16:44:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.648+0000 7f5c13e00640 1 -- 172.21.15.28:0/342719926 wait complete. 2024-09-13T16:44:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.649+0000 7f5c13e00640 1 Processor -- start 2024-09-13T16:44:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.649+0000 7f5c13e00640 1 -- start start 2024-09-13T16:44:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.650+0000 7f5c13e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c140c1b40 0x7f5c141f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.650+0000 7f5c13e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c140c6240 0x7f5c141536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.650+0000 7f5c13e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c141509b0 0x7f5c14157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.650+0000 7f5c13e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1415abd0 con 0x7f5c141509b0 2024-09-13T16:44:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.650+0000 7f5c13e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1414b220 con 0x7f5c140c6240 2024-09-13T16:44:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.650+0000 7f5c13e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c14157a70 con 0x7f5c140c1b40 2024-09-13T16:44:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.651+0000 7f5c13400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c141509b0 0x7f5c14157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.651+0000 7f5c12000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c140c6240 0x7f5c141536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.651+0000 7f5c12a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c140c1b40 0x7f5c141f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.651+0000 7f5c12000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c140c6240 0x7f5c141536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:50698/0 (socket says 172.21.15.28:50698) 2024-09-13T16:44:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.651+0000 7f5c12000640 1 -- 172.21.15.28:0/1233205410 learned_addr learned my addr 172.21.15.28:0/1233205410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.651+0000 7f5c13400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c141509b0 0x7f5c14157f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60278/0 (socket says 172.21.15.28:60278) 2024-09-13T16:44:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.651+0000 7f5c13400640 1 -- 172.21.15.28:0/1233205410 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c140c1b40 msgr2=0x7f5c141f27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.651+0000 7f5c13400640 1 --2- 172.21.15.28:0/1233205410 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c140c1b40 0x7f5c141f27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.651+0000 7f5c13400640 1 -- 172.21.15.28:0/1233205410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c140c6240 msgr2=0x7f5c141536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.651+0000 7f5c13400640 1 --2- 172.21.15.28:0/1233205410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c140c6240 0x7f5c141536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.651+0000 7f5c13400640 1 -- 172.21.15.28:0/1233205410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c00009400 con 0x7f5c141509b0 2024-09-13T16:44:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.651+0000 7f5c12000640 1 --2- 172.21.15.28:0/1233205410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c140c6240 0x7f5c141536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.651+0000 7f5c13400640 1 --2- 172.21.15.28:0/1233205410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c141509b0 0x7f5c14157f00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5c1406af40 tx=0x7f5c0c00cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.652+0000 7f5c0aa00640 1 -- 172.21.15.28:0/1233205410 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c004d60 con 0x7f5c141509b0 2024-09-13T16:44:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.652+0000 7f5c0aa00640 1 -- 172.21.15.28:0/1233205410 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0c00a640 con 0x7f5c141509b0 2024-09-13T16:44:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.652+0000 7f5c0aa00640 1 -- 172.21.15.28:0/1233205410 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c00a9f0 con 0x7f5c141509b0 2024-09-13T16:44:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.652+0000 7f5c13e00640 1 -- 172.21.15.28:0/1233205410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c04009400 con 0x7f5c141509b0 2024-09-13T16:44:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.652+0000 7f5c13e00640 1 -- 172.21.15.28:0/1233205410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c14159ce0 con 0x7f5c141509b0 2024-09-13T16:44:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.653+0000 7f5c13e00640 1 -- 172.21.15.28:0/1233205410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c14157c20 con 0x7f5c141509b0 2024-09-13T16:44:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.654+0000 7f5c0aa00640 1 -- 172.21.15.28:0/1233205410 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5c0c00ab50 con 0x7f5c141509b0 2024-09-13T16:44:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.655+0000 7f5c0aa00640 1 --2- 172.21.15.28:0/1233205410 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f5bf406ac00 0x7f5bf406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.655+0000 7f5c0aa00640 1 -- 172.21.15.28:0/1233205410 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c0c088710 con 0x7f5c141509b0 2024-09-13T16:44:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.655+0000 7f5c12a00640 1 --2- 172.21.15.28:0/1233205410 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f5bf406ac00 0x7f5bf406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:40.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.656+0000 7f5c12a00640 1 --2- 172.21.15.28:0/1233205410 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f5bf406ac00 0x7f5bf406d0c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5c04034000 tx=0x7f5c0402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:40.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.660+0000 7f5c0aa00640 1 -- 172.21.15.28:0/1233205410 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5c14157c20 con 0x7f5c141509b0 2024-09-13T16:44:40.667 INFO:tasks.workunit.client.0.smithi028.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-13T16:44:40.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.883+0000 7f5c13e00640 1 -- 172.21.15.28:0/1233205410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c14157c20 con 0x7f5c141509b0 2024-09-13T16:44:40.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.883+0000 7f5c0aa00640 1 -- 172.21.15.28:0/1233205410 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6307 (secure 0 0 0) 0x7f5c14157c20 con 0x7f5c141509b0 2024-09-13T16:44:40.886 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:44:40.886 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4227,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/355570369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":355570369},{"type":"v1","addr":"172.21.15.28:6835","nonce":355570369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:43:54.969699+0000","modified":"2024-09-13T16:44:26.685528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4250,"mds_2":4244,"mds_3":4235,"mds_4":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"f","rank":4,"incarnation":16,"state":"up:active","state_seq":10,"addr":"172.21.15.191:6839/2366383263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6837","nonce":2366383263},{"type":"v1","addr":"172.21.15.191:6839","nonce":2366383263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4235":{"gid":4235,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.191:6838/2702860767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2702860767},{"type":"v1","addr":"172.21.15.191:6838","nonce":2702860767}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/2240589617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2240589617},{"type":"v1","addr":"172.21.15.28:6833","nonce":2240589617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6837/4068563017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":4068563017},{"type":"v1","addr":"172.21.15.28:6837","nonce":4068563017}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4258":{"gid":4258,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/704943585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":704943585},{"type":"v1","addr":"172.21.15.191:6835","nonce":704943585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T16:44:40.888 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-09-13T16:44:40.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.886+0000 7f5c13e00640 1 -- 172.21.15.28:0/1233205410 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f5bf406ac00 msgr2=0x7f5bf406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:40.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.886+0000 7f5c13e00640 1 --2- 172.21.15.28:0/1233205410 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f5bf406ac00 0x7f5bf406d0c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5c04034000 tx=0x7f5c0402e000 comp rx=0 tx=0).stop 2024-09-13T16:44:40.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.886+0000 7f5c13e00640 1 -- 172.21.15.28:0/1233205410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c141509b0 msgr2=0x7f5c14157f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:40.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.886+0000 7f5c13e00640 1 --2- 172.21.15.28:0/1233205410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c141509b0 0x7f5c14157f00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5c1406af40 tx=0x7f5c0c00cb60 comp rx=0 tx=0).stop 2024-09-13T16:44:40.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.886+0000 7f5c13e00640 1 -- 172.21.15.28:0/1233205410 shutdown_connections 2024-09-13T16:44:40.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.886+0000 7f5c13e00640 1 --2- 172.21.15.28:0/1233205410 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f5bf406ac00 0x7f5bf406d0c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:40.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.886+0000 7f5c13e00640 1 --2- 172.21.15.28:0/1233205410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c141509b0 0x7f5c14157f00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:40.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.886+0000 7f5c13e00640 1 --2- 172.21.15.28:0/1233205410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5c140c6240 0x7f5c141536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:40.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.886+0000 7f5c13e00640 1 --2- 172.21.15.28:0/1233205410 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5c140c1b40 0x7f5c141f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:40.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.886+0000 7f5c13e00640 1 -- 172.21.15.28:0/1233205410 >> 172.21.15.28:0/1233205410 conn(0x7f5c140ca0e0 msgr2=0x7f5c140c1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:40.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.886+0000 7f5c13e00640 1 -- 172.21.15.28:0/1233205410 shutdown_connections 2024-09-13T16:44:40.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:40.886+0000 7f5c13e00640 1 -- 172.21.15.28:0/1233205410 wait complete. 2024-09-13T16:44:40.904 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-13T16:44:40.904 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726245878, revive delay: 20.0, thrash delay: 10.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-13T16:44:40.904 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T16:44:40.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:44:41.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.097+0000 7f01a9800640 1 Processor -- start 2024-09-13T16:44:41.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.097+0000 7f01a9800640 1 -- start start 2024-09-13T16:44:41.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.098+0000 7f01a9800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f01a4152cc0 0x7f01a40c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.098+0000 7f01a9800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01a41530a0 0x7f01a40c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.098+0000 7f01a9800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a40c5e70 con 0x7f01a40c6570 2024-09-13T16:44:41.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.098+0000 7f01a9800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a406d4c0 con 0x7f01a4152cc0 2024-09-13T16:44:41.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.098+0000 7f01a9800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a415c670 con 0x7f01a41530a0 2024-09-13T16:44:41.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.099+0000 7f01a2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f01a4152cc0 0x7f01a40c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.099+0000 7f01a3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f01a40c6570 0x7f01a406d950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57200/0 (socket says 172.21.15.28:57200) 2024-09-13T16:44:41.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.099+0000 7f01a3400640 1 -- 172.21.15.28:0/3539624228 learned_addr learned my addr 172.21.15.28:0/3539624228 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:41.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.099+0000 7f01a2000640 1 --2- 172.21.15.28:0/3539624228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01a41530a0 0x7f01a40c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.099+0000 7f01a2a00640 1 -- 172.21.15.28:0/3539624228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01a41530a0 msgr2=0x7f01a40c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:41.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.099+0000 7f01a2a00640 1 --2- 172.21.15.28:0/3539624228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01a41530a0 0x7f01a40c7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.099+0000 7f01a2a00640 1 -- 172.21.15.28:0/3539624228 >> v1:172.21.15.28:6789/0 conn(0x7f01a40c6570 legacy=0x7f01a406d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.099+0000 7f01a1600640 1 -- 172.21.15.28:0/3539624228 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192961729 0 0) 0x7f01a40c5e70 con 0x7f01a40c6570 2024-09-13T16:44:41.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.099+0000 7f01a2a00640 1 -- 172.21.15.28:0/3539624228 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a415c950 con 0x7f01a4152cc0 2024-09-13T16:44:41.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.100+0000 7f01a2a00640 1 --2- 172.21.15.28:0/3539624228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01a4152cc0 0x7f01a40c6ef0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f01940097a0 tx=0x7f019402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fb42ee2fe146ad9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.101+0000 7f01a1600640 1 -- 172.21.15.28:0/3539624228 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0194032440 con 0x7f01a4152cc0 2024-09-13T16:44:41.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.101+0000 7f01a1600640 1 -- 172.21.15.28:0/3539624228 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01940325a0 con 0x7f01a4152cc0 2024-09-13T16:44:41.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.101+0000 7f01a1600640 1 -- 172.21.15.28:0/3539624228 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0194032930 con 0x7f01a4152cc0 2024-09-13T16:44:41.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.101+0000 7f01a9800640 1 -- 172.21.15.28:0/3539624228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01a4152cc0 msgr2=0x7f01a40c6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.101+0000 7f01a9800640 1 --2- 172.21.15.28:0/3539624228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01a4152cc0 0x7f01a40c6ef0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f01940097a0 tx=0x7f019402a210 comp rx=0 tx=0).stop 2024-09-13T16:44:41.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.101+0000 7f01a9800640 1 -- 172.21.15.28:0/3539624228 shutdown_connections 2024-09-13T16:44:41.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.101+0000 7f01a9800640 1 --2- 172.21.15.28:0/3539624228 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01a41530a0 0x7f01a40c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.101+0000 7f01a9800640 1 --2- 172.21.15.28:0/3539624228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01a4152cc0 0x7f01a40c6ef0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.101+0000 7f01a9800640 1 -- 172.21.15.28:0/3539624228 >> 172.21.15.28:0/3539624228 conn(0x7f01a4069e80 msgr2=0x7f01a406a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:41.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.102+0000 7f01a9800640 1 -- 172.21.15.28:0/3539624228 shutdown_connections 2024-09-13T16:44:41.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.102+0000 7f01a9800640 1 -- 172.21.15.28:0/3539624228 wait complete. 2024-09-13T16:44:41.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.103+0000 7f01a9800640 1 Processor -- start 2024-09-13T16:44:41.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.103+0000 7f01a9800640 1 -- start start 2024-09-13T16:44:41.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.104+0000 7f01a9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01a40c6570 0x7f01a40c4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.104+0000 7f01a9800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01a4152cc0 0x7f01a41e5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.104+0000 7f01a9800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01a41530a0 0x7f01a41e5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.104+0000 7f01a9800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a41e2450 con 0x7f01a40c6570 2024-09-13T16:44:41.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.104+0000 7f01a9800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a4156500 con 0x7f01a41530a0 2024-09-13T16:44:41.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.104+0000 7f01a9800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a41ef590 con 0x7f01a4152cc0 2024-09-13T16:44:41.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.104+0000 7f01a2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01a40c6570 0x7f01a40c4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.104+0000 7f01a2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01a40c6570 0x7f01a40c4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60296/0 (socket says 172.21.15.28:60296) 2024-09-13T16:44:41.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.104+0000 7f01a2a00640 1 -- 172.21.15.28:0/1783357363 learned_addr learned my addr 172.21.15.28:0/1783357363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:41.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.105+0000 7f01a2000640 1 --2- 172.21.15.28:0/1783357363 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01a4152cc0 0x7f01a41e5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.105+0000 7f01a3400640 1 --2- 172.21.15.28:0/1783357363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01a41530a0 0x7f01a41e5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.105+0000 7f01a2a00640 1 -- 172.21.15.28:0/1783357363 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01a4152cc0 msgr2=0x7f01a41e5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.105+0000 7f01a2a00640 1 --2- 172.21.15.28:0/1783357363 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01a4152cc0 0x7f01a41e5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.105+0000 7f01a2a00640 1 -- 172.21.15.28:0/1783357363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01a41530a0 msgr2=0x7f01a41e5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.105+0000 7f01a2a00640 1 --2- 172.21.15.28:0/1783357363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01a41530a0 0x7f01a41e5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.105+0000 7f01a2a00640 1 -- 172.21.15.28:0/1783357363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0194009400 con 0x7f01a40c6570 2024-09-13T16:44:41.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.105+0000 7f01a2a00640 1 --2- 172.21.15.28:0/1783357363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01a40c6570 0x7f01a40c4dc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0194000c00 tx=0x7f01940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.105+0000 7f019aa00640 1 -- 172.21.15.28:0/1783357363 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0194030b80 con 0x7f01a40c6570 2024-09-13T16:44:41.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.106+0000 7f019aa00640 1 -- 172.21.15.28:0/1783357363 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0194032e90 con 0x7f01a40c6570 2024-09-13T16:44:41.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.106+0000 7f01a9800640 1 -- 172.21.15.28:0/1783357363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a4155f40 con 0x7f01a40c6570 2024-09-13T16:44:41.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.106+0000 7f019aa00640 1 -- 172.21.15.28:0/1783357363 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0194006e10 con 0x7f01a40c6570 2024-09-13T16:44:41.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.106+0000 7f01a9800640 1 -- 172.21.15.28:0/1783357363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a41ef830 con 0x7f01a40c6570 2024-09-13T16:44:41.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.107+0000 7f01a9800640 1 -- 172.21.15.28:0/1783357363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a40c6d60 con 0x7f01a40c6570 2024-09-13T16:44:41.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.108+0000 7f019aa00640 1 -- 172.21.15.28:0/1783357363 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0194039040 con 0x7f01a40c6570 2024-09-13T16:44:41.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.108+0000 7f019aa00640 1 --2- 172.21.15.28:0/1783357363 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f018406ac00 0x7f018406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.109+0000 7f019aa00640 1 -- 172.21.15.28:0/1783357363 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01940a9ce0 con 0x7f01a40c6570 2024-09-13T16:44:41.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.109+0000 7f01a2000640 1 --2- 172.21.15.28:0/1783357363 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f018406ac00 0x7f018406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.109+0000 7f01a2000640 1 --2- 172.21.15.28:0/1783357363 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f018406ac00 0x7f018406d0c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f01a4071db0 tx=0x7f0190009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.112+0000 7f019aa00640 1 -- 172.21.15.28:0/1783357363 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0194074ed0 con 0x7f01a40c6570 2024-09-13T16:44:41.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.363+0000 7f01a9800640 1 -- 172.21.15.28:0/1783357363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f01a40c6020 con 0x7f01a40c6570 2024-09-13T16:44:41.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.364+0000 7f019aa00640 1 -- 172.21.15.28:0/1783357363 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7f0194037620 con 0x7f01a40c6570 2024-09-13T16:44:41.366 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":61,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:44:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.366+0000 7f01a9800640 1 -- 172.21.15.28:0/1783357363 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f018406ac00 msgr2=0x7f018406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.366+0000 7f01a9800640 1 --2- 172.21.15.28:0/1783357363 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f018406ac00 0x7f018406d0c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f01a4071db0 tx=0x7f0190009190 comp rx=0 tx=0).stop 2024-09-13T16:44:41.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.366+0000 7f01a9800640 1 -- 172.21.15.28:0/1783357363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01a40c6570 msgr2=0x7f01a40c4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.366+0000 7f01a9800640 1 --2- 172.21.15.28:0/1783357363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01a40c6570 0x7f01a40c4dc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0194000c00 tx=0x7f01940046c0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.366+0000 7f01a9800640 1 -- 172.21.15.28:0/1783357363 shutdown_connections 2024-09-13T16:44:41.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.366+0000 7f01a9800640 1 --2- 172.21.15.28:0/1783357363 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f018406ac00 0x7f018406d0c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.366+0000 7f01a9800640 1 --2- 172.21.15.28:0/1783357363 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01a41530a0 0x7f01a41e5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.366+0000 7f01a9800640 1 --2- 172.21.15.28:0/1783357363 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01a4152cc0 0x7f01a41e5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.366+0000 7f01a9800640 1 --2- 172.21.15.28:0/1783357363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01a40c6570 0x7f01a40c4dc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.366+0000 7f01a9800640 1 -- 172.21.15.28:0/1783357363 >> 172.21.15.28:0/1783357363 conn(0x7f01a4069e80 msgr2=0x7f01a40c92b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:41.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.366+0000 7f01a9800640 1 -- 172.21.15.28:0/1783357363 shutdown_connections 2024-09-13T16:44:41.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.366+0000 7f01a9800640 1 -- 172.21.15.28:0/1783357363 wait complete. 2024-09-13T16:44:41.384 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T16:44:41.384 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-13T16:44:41.385 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-13T16:44:41.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T16:44:41.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.539+0000 7fd4c7e00640 1 Processor -- start 2024-09-13T16:44:41.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.539+0000 7fd4c7e00640 1 -- start start 2024-09-13T16:44:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.540+0000 7fd4c7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd4c00c5140 0x7fd4c0159800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.540+0000 7fd4c7e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4c00c7830 0x7fd4c014f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.540+0000 7fd4c7e00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c00c6f50 con 0x7fd4c00c73e0 2024-09-13T16:44:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.540+0000 7fd4c7e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c00c6940 con 0x7fd4c00c5140 2024-09-13T16:44:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.540+0000 7fd4c7e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c0159d40 con 0x7fd4c00c7830 2024-09-13T16:44:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.541+0000 7fd4c5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd4c00c5140 0x7fd4c0159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.541+0000 7fd4c5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd4c00c5140 0x7fd4c0159800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60300/0 (socket says 172.21.15.28:60300) 2024-09-13T16:44:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.541+0000 7fd4c5400640 1 -- 172.21.15.28:0/1784126504 learned_addr learned my addr 172.21.15.28:0/1784126504 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.541+0000 7fd4c5e00640 1 --2- 172.21.15.28:0/1784126504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4c00c7830 0x7fd4c014f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.541+0000 7fd4c5400640 1 -- 172.21.15.28:0/1784126504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4c00c7830 msgr2=0x7fd4c014f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.541+0000 7fd4bfe00640 1 -- 172.21.15.28:0/1784126504 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001896913 0 0) 0x7fd4c00c6f50 con 0x7fd4c00c73e0 2024-09-13T16:44:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.541+0000 7fd4c5400640 1 --2- 172.21.15.28:0/1784126504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4c00c7830 0x7fd4c014f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.541+0000 7fd4c5400640 1 -- 172.21.15.28:0/1784126504 >> v1:172.21.15.191:6789/0 conn(0x7fd4c00c73e0 legacy=0x7fd4c00c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.541+0000 7fd4c5400640 1 -- 172.21.15.28:0/1784126504 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c00c5c30 con 0x7fd4c00c5140 2024-09-13T16:44:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.541+0000 7fd4c5e00640 1 --2- 172.21.15.28:0/1784126504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4c00c7830 0x7fd4c014f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.542+0000 7fd4c5400640 1 --2- 172.21.15.28:0/1784126504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c00c5140 0x7fd4c0159800 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4009870 tx=0x7fd4b402c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=560d797dc511f74a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.542+0000 7fd4bfe00640 1 -- 172.21.15.28:0/1784126504 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4031710 con 0x7fd4c00c5140 2024-09-13T16:44:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.542+0000 7fd4bfe00640 1 -- 172.21.15.28:0/1784126504 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b402f050 con 0x7fd4c00c5140 2024-09-13T16:44:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.542+0000 7fd4bfe00640 1 -- 172.21.15.28:0/1784126504 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4031a90 con 0x7fd4c00c5140 2024-09-13T16:44:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.543+0000 7fd4c7e00640 1 -- 172.21.15.28:0/1784126504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c00c5140 msgr2=0x7fd4c0159800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.543+0000 7fd4c7e00640 1 --2- 172.21.15.28:0/1784126504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c00c5140 0x7fd4c0159800 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4009870 tx=0x7fd4b402c3d0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.543+0000 7fd4c7e00640 1 -- 172.21.15.28:0/1784126504 shutdown_connections 2024-09-13T16:44:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.543+0000 7fd4c7e00640 1 --2- 172.21.15.28:0/1784126504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4c00c7830 0x7fd4c014f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.543+0000 7fd4c7e00640 1 --2- 172.21.15.28:0/1784126504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c00c5140 0x7fd4c0159800 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.543+0000 7fd4c7e00640 1 -- 172.21.15.28:0/1784126504 >> 172.21.15.28:0/1784126504 conn(0x7fd4c00ca6b0 msgr2=0x7fd4c00caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.543+0000 7fd4c7e00640 1 -- 172.21.15.28:0/1784126504 shutdown_connections 2024-09-13T16:44:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.543+0000 7fd4c7e00640 1 -- 172.21.15.28:0/1784126504 wait complete. 2024-09-13T16:44:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.545+0000 7fd4c7e00640 1 Processor -- start 2024-09-13T16:44:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.545+0000 7fd4c7e00640 1 -- start start 2024-09-13T16:44:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.546+0000 7fd4c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c00c5140 0x7fd4c01545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.546+0000 7fd4c7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4c00c73e0 0x7fd4c014a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.546+0000 7fd4c7e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4c0070cf0 0x7fd4c0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.546+0000 7fd4c7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c014d4d0 con 0x7fd4c00c5140 2024-09-13T16:44:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.546+0000 7fd4c7e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c01565a0 con 0x7fd4c00c73e0 2024-09-13T16:44:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.546+0000 7fd4c7e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c0154b00 con 0x7fd4c0070cf0 2024-09-13T16:44:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.546+0000 7fd4c5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c00c5140 0x7fd4c01545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.546+0000 7fd4c5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c00c5140 0x7fd4c01545c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60306/0 (socket says 172.21.15.28:60306) 2024-09-13T16:44:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.546+0000 7fd4c5400640 1 -- 172.21.15.28:0/2868104045 learned_addr learned my addr 172.21.15.28:0/2868104045 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.546+0000 7fd4c4a00640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4c00c73e0 0x7fd4c014a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.546+0000 7fd4c5e00640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4c0070cf0 0x7fd4c0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.547+0000 7fd4c5400640 1 -- 172.21.15.28:0/2868104045 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4c0070cf0 msgr2=0x7fd4c0071180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.547+0000 7fd4c5400640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4c0070cf0 0x7fd4c0071180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.547+0000 7fd4c5400640 1 -- 172.21.15.28:0/2868104045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4c00c73e0 msgr2=0x7fd4c014a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.547+0000 7fd4c5400640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4c00c73e0 0x7fd4c014a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.547+0000 7fd4c5400640 1 -- 172.21.15.28:0/2868104045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b8009a70 con 0x7fd4c00c5140 2024-09-13T16:44:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.547+0000 7fd4c5e00640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4c0070cf0 0x7fd4c0071180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.547+0000 7fd4c5400640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c00c5140 0x7fd4c01545c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4000c00 tx=0x7fd4b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.547+0000 7fd4bd600640 1 -- 172.21.15.28:0/2868104045 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b403c830 con 0x7fd4c00c5140 2024-09-13T16:44:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.547+0000 7fd4bd600640 1 -- 172.21.15.28:0/2868104045 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b403c990 con 0x7fd4c00c5140 2024-09-13T16:44:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.547+0000 7fd4c7e00640 1 -- 172.21.15.28:0/2868104045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b40094d0 con 0x7fd4c00c5140 2024-09-13T16:44:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.547+0000 7fd4bd600640 1 -- 172.21.15.28:0/2868104045 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b403cd40 con 0x7fd4c00c5140 2024-09-13T16:44:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.547+0000 7fd4c7e00640 1 -- 172.21.15.28:0/2868104045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c0155fe0 con 0x7fd4c00c5140 2024-09-13T16:44:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.548+0000 7fd4c7e00640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c00c7830 0x7fd4c0070390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.548+0000 7fd4c7e00640 1 -- 172.21.15.28:0/2868104045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c0066c20 con 0x7fd4c00c7830 2024-09-13T16:44:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.549+0000 7fd4c4a00640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c00c7830 0x7fd4c0070390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.549+0000 7fd4c4a00640 1 -- 172.21.15.28:0/2868104045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4c00c6be0 con 0x7fd4c00c7830 2024-09-13T16:44:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.549+0000 7fd4c4a00640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c00c7830 0x7fd4c0070390 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd4c01497e0 tx=0x7fd4ac00fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.549+0000 7fd4bd600640 1 -- 172.21.15.28:0/2868104045 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd4b403a070 con 0x7fd4c00c5140 2024-09-13T16:44:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.550+0000 7fd4bd600640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd4a006ac30 0x7fd4a006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.550+0000 7fd4bd600640 1 -- 172.21.15.28:0/2868104045 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4b40a9f90 con 0x7fd4c00c5140 2024-09-13T16:44:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.550+0000 7fd4bd600640 1 -- 172.21.15.28:0/2868104045 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac0050d0 con 0x7fd4c00c7830 2024-09-13T16:44:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.550+0000 7fd4bd600640 1 -- 172.21.15.28:0/2868104045 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fd4ac005230 con 0x7fd4c00c7830 2024-09-13T16:44:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.550+0000 7fd4bd600640 1 -- 172.21.15.28:0/2868104045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c00c7830 msgr2=0x7fd4c0070390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.550+0000 7fd4bd600640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c00c7830 0x7fd4c0070390 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd4c01497e0 tx=0x7fd4ac00fc00 comp rx=0 tx=0).stop 2024-09-13T16:44:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.550+0000 7fd4bd600640 1 -- 172.21.15.28:0/2868104045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c00c7830 msgr2=0x7fd4c0070390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:44:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.550+0000 7fd4bd600640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c00c7830 0x7fd4c0070390 secure :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd4c01497e0 tx=0x7fd4ac00fc00 comp rx=0 tx=0).stop 2024-09-13T16:44:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.550+0000 7fd4c5e00640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd4a006ac30 0x7fd4a006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.551+0000 7fd4c5e00640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd4a006ac30 0x7fd4a006d0f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8009fd0 tx=0x7fd4b800b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.557+0000 7fd4c7e00640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c0140c70 0x7fd4c0141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.557+0000 7fd4c7e00640 1 -- 172.21.15.28:0/2868104045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c00c1e30 con 0x7fd4c0140c70 2024-09-13T16:44:41.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.557+0000 7fd4c4a00640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c0140c70 0x7fd4c0141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.558+0000 7fd4c4a00640 1 -- 172.21.15.28:0/2868104045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd4c00c2110 con 0x7fd4c0140c70 2024-09-13T16:44:41.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.558+0000 7fd4c4a00640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c0140c70 0x7fd4c0141080 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00c23f0 tx=0x7fd4ac001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.558+0000 7fd4bd600640 1 -- 172.21.15.28:0/2868104045 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac023070 con 0x7fd4c0140c70 2024-09-13T16:44:41.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.558+0000 7fd4bd600640 1 -- 172.21.15.28:0/2868104045 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4934 (secure 0 0 0) 0x7fd4ac00dbe0 con 0x7fd4c0140c70 2024-09-13T16:44:41.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.558+0000 7fd4bd600640 1 -- 172.21.15.28:0/2868104045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c0140c70 msgr2=0x7fd4c0141080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.558+0000 7fd4bd600640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c0140c70 0x7fd4c0141080 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00c23f0 tx=0x7fd4ac001080 comp rx=0 tx=0).stop 2024-09-13T16:44:41.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.558+0000 7fd4bd600640 1 -- 172.21.15.28:0/2868104045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c0140c70 msgr2=0x7fd4c0141080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:44:41.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.558+0000 7fd4bd600640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c0140c70 0x7fd4c0141080 secure :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00c23f0 tx=0x7fd4ac001080 comp rx=0 tx=0).stop 2024-09-13T16:44:41.560 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:44:41.561 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:44:41.561 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:44:41.561 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "leader", 2024-09-13T16:44:41.561 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 4, 2024-09-13T16:44:41.561 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:44:41.561 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:44:41.561 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:44:41.561 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:44:41.561 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.561 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 62, 2024-09-13T16:44:41.561 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:44:41.561 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:44:41.562 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:44:41.562 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:44:41.562 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:44:41.562 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:44:41.562 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:44:41.562 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:44:41.562 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:44:41.562 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:44:41.562 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:44:41.562 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:44:41.562 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:44:41.562 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.562 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:44:41.563 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:44:41.563 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:44:41.563 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:44:41.563 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:44:41.563 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:44:41.563 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:44:41.563 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:44:41.563 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:44:41.563 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:44:41.563 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:44:41.563 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:44:41.563 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.564 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:44:41.564 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:44:41.564 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:44:41.564 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:44:41.564 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:44:41.564 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:44:41.564 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:44:41.564 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:44:41.564 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:44:41.564 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:44:41.564 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:44:41.564 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:44:41.564 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:44:41.565 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:44:41.565 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:44:41.565 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:44:41.565 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:44:41.565 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:44:41.565 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:44:41.565 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:44:41.565 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:44:41.565 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:44:41.565 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:44:41.565 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:44:41.565 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:44:41.565 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:44:41.565 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:44:41.566 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.566 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:44:41.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.566 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:44:41.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.566 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:44:41.566 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:44:41.566 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:44:41.566 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:44:41.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.566 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:44:41.566 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:44:41.566 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.567 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.567 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:44:41.567 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:44:41.567 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.567 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.567 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.567 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:44:41.567 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:44:41.567 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:44:41.567 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:44:41.567 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:44:41.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.568 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:44:41.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:44:41.568 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:44:41.568 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:44:41.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.568 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:44:41.568 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:44:41.568 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.568 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:44:41.568 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:44:41.568 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.569 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.569 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.569 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:44:41.569 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:44:41.569 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:44:41.569 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:44:41.569 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:44:41.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.569 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:44:41.569 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:44:41.569 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:44:41.570 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:44:41.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.570 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:44:41.570 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:44:41.570 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.570 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:44:41.570 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:44:41.570 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.570 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.570 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.570 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:44:41.571 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:44:41.571 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:44:41.571 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:44:41.571 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:44:41.571 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.571 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.571 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:44:41.571 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:44:41.571 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.571 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:41.571 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:41.571 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:44:41.571 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.572 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.572 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:44:41.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.572 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:41.572 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:41.572 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:44:41.572 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.572 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.572 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:44:41.572 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.572 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:41.572 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:41.572 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:44:41.572 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.573 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.573 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:44:41.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.573 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x2f018fb87aa4aafe", 2024-09-13T16:44:41.573 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "luminous", 2024-09-13T16:44:41.573 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:44:41.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.573 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:41.573 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:41.573 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:44:41.573 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.573 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.574 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:44:41.574 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:44:41.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.559+0000 7fd4c7e00640 1 -- 172.21.15.28:0/2868104045 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd4a006ac30 msgr2=0x7fd4a006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.559+0000 7fd4c7e00640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd4a006ac30 0x7fd4a006d0f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8009fd0 tx=0x7fd4b800b040 comp rx=0 tx=0).stop 2024-09-13T16:44:41.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.559+0000 7fd4c7e00640 1 -- 172.21.15.28:0/2868104045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c00c5140 msgr2=0x7fd4c01545c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.559+0000 7fd4c7e00640 1 --2- 172.21.15.28:0/2868104045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c00c5140 0x7fd4c01545c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4000c00 tx=0x7fd4b40046c0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.559+0000 7fd4c5e00640 1 -- 172.21.15.28:0/2868104045 reap_dead start 2024-09-13T16:44:41.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.559+0000 7fd4c5e00640 1 -- 172.21.15.28:0/2868104045 reap_dead start 2024-09-13T16:44:41.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.559+0000 7fd4c7e00640 1 -- 172.21.15.28:0/2868104045 shutdown_connections 2024-09-13T16:44:41.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.559+0000 7fd4c7e00640 1 -- 172.21.15.28:0/2868104045 >> 172.21.15.28:0/2868104045 conn(0x7fd4c00ca6b0 msgr2=0x7fd4c0144260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:41.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.559+0000 7fd4c7e00640 1 -- 172.21.15.28:0/2868104045 shutdown_connections 2024-09-13T16:44:41.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.559+0000 7fd4c7e00640 1 -- 172.21.15.28:0/2868104045 wait complete. 2024-09-13T16:44:41.576 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:44:41.576 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-13T16:44:41.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T16:44:41.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.731+0000 7ff9e6a00640 1 Processor -- start 2024-09-13T16:44:41.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.731+0000 7ff9e6a00640 1 -- start start 2024-09-13T16:44:41.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.733+0000 7ff9e6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff9e0156f50 0x7ff9e00c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.733+0000 7ff9e6a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e0150d70 con 0x7ff9e00c17f0 2024-09-13T16:44:41.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.733+0000 7ff9e6a00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e00c5310 con 0x7ff9e0150990 2024-09-13T16:44:41.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.733+0000 7ff9e6a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e00c2190 con 0x7ff9e0156f50 2024-09-13T16:44:41.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.733+0000 7ff9e6000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff9e0156f50 0x7ff9e00c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.733+0000 7ff9e4c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff9e00c17f0 0x7ff9e0156840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57210/0 (socket says 172.21.15.28:57210) 2024-09-13T16:44:41.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.733+0000 7ff9e4c00640 1 -- 172.21.15.28:0/4277224407 learned_addr learned my addr 172.21.15.28:0/4277224407 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:41.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.733+0000 7ff9e5600640 1 --1- 172.21.15.28:0/4277224407 >> v1:172.21.15.191:6789/0 conn(0x7ff9e0150990 0x7ff9e006b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:51410/0 (socket says 172.21.15.28:51410) 2024-09-13T16:44:41.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.734+0000 7ff9e6000640 1 -- 172.21.15.28:0/4277224407 >> v1:172.21.15.191:6789/0 conn(0x7ff9e0150990 legacy=0x7ff9e006b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.734+0000 7ff9e6000640 1 -- 172.21.15.28:0/4277224407 >> v1:172.21.15.28:6789/0 conn(0x7ff9e00c17f0 legacy=0x7ff9e0156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.734+0000 7ff9e6000640 1 -- 172.21.15.28:0/4277224407 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e00c2470 con 0x7ff9e0156f50 2024-09-13T16:44:41.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.735+0000 7ff9e6000640 1 --2- 172.21.15.28:0/4277224407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e0156f50 0x7ff9e00c4cf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8009d90 tx=0x7ff9d802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a6b6d833c4cb00 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.735+0000 7ff9dfe00640 1 -- 172.21.15.28:0/4277224407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8007740 con 0x7ff9e0156f50 2024-09-13T16:44:41.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.735+0000 7ff9dfe00640 1 -- 172.21.15.28:0/4277224407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d802d050 con 0x7ff9e0156f50 2024-09-13T16:44:41.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.735+0000 7ff9dfe00640 1 -- 172.21.15.28:0/4277224407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8007ac0 con 0x7ff9e0156f50 2024-09-13T16:44:41.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.735+0000 7ff9e6a00640 1 -- 172.21.15.28:0/4277224407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e0156f50 msgr2=0x7ff9e00c4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.735+0000 7ff9e6a00640 1 --2- 172.21.15.28:0/4277224407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e0156f50 0x7ff9e00c4cf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8009d90 tx=0x7ff9d802a6f0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.735+0000 7ff9e6a00640 1 -- 172.21.15.28:0/4277224407 shutdown_connections 2024-09-13T16:44:41.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.735+0000 7ff9e6a00640 1 --2- 172.21.15.28:0/4277224407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e0156f50 0x7ff9e00c4cf0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.735+0000 7ff9e6a00640 1 -- 172.21.15.28:0/4277224407 >> 172.21.15.28:0/4277224407 conn(0x7ff9e00ca2f0 msgr2=0x7ff9e00ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:41.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.736+0000 7ff9e6a00640 1 -- 172.21.15.28:0/4277224407 shutdown_connections 2024-09-13T16:44:41.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.736+0000 7ff9e6a00640 1 -- 172.21.15.28:0/4277224407 wait complete. 2024-09-13T16:44:41.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.737+0000 7ff9e6a00640 1 Processor -- start 2024-09-13T16:44:41.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.737+0000 7ff9e6a00640 1 -- start start 2024-09-13T16:44:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.738+0000 7ff9e6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e00c17f0 0x7ff9e01ef230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.738+0000 7ff9e6a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e0150990 0x7ff9e01e5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.738+0000 7ff9e6a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9e0156f50 0x7ff9e01e5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.738+0000 7ff9e6a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e0070f90 con 0x7ff9e00c17f0 2024-09-13T16:44:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.738+0000 7ff9e6a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e0070bf0 con 0x7ff9e0150990 2024-09-13T16:44:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.738+0000 7ff9e6a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e014a820 con 0x7ff9e0156f50 2024-09-13T16:44:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.738+0000 7ff9e5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e00c17f0 0x7ff9e01ef230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.738+0000 7ff9e5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e00c17f0 0x7ff9e01ef230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60346/0 (socket says 172.21.15.28:60346) 2024-09-13T16:44:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.738+0000 7ff9e5600640 1 -- 172.21.15.28:0/4047746761 learned_addr learned my addr 172.21.15.28:0/4047746761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.739+0000 7ff9e6000640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9e0156f50 0x7ff9e01e5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.739+0000 7ff9e4c00640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e0150990 0x7ff9e01e5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.739+0000 7ff9e5600640 1 -- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9e0156f50 msgr2=0x7ff9e01e5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.739+0000 7ff9e5600640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9e0156f50 0x7ff9e01e5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.739+0000 7ff9e5600640 1 -- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e0150990 msgr2=0x7ff9e01e5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.739+0000 7ff9e5600640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e0150990 0x7ff9e01e5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.739+0000 7ff9e5600640 1 -- 172.21.15.28:0/4047746761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d8009a70 con 0x7ff9e00c17f0 2024-09-13T16:44:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.739+0000 7ff9e5600640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e00c17f0 0x7ff9e01ef230 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400c3e0 tx=0x7ff9d400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.739+0000 7ff9dd600640 1 -- 172.21.15.28:0/4047746761 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d40074e0 con 0x7ff9e00c17f0 2024-09-13T16:44:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.739+0000 7ff9dd600640 1 -- 172.21.15.28:0/4047746761 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d4007640 con 0x7ff9e00c17f0 2024-09-13T16:44:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.739+0000 7ff9dd600640 1 -- 172.21.15.28:0/4047746761 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d40049e0 con 0x7ff9e00c17f0 2024-09-13T16:44:41.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.739+0000 7ff9e6a00640 1 -- 172.21.15.28:0/4047746761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e0070910 con 0x7ff9e00c17f0 2024-09-13T16:44:41.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.740+0000 7ff9e6a00640 1 -- 172.21.15.28:0/4047746761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e01eea30 con 0x7ff9e00c17f0 2024-09-13T16:44:41.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.740+0000 7ff9e6a00640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e00c81a0 0x7ff9e01eebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.740+0000 7ff9e6a00640 1 -- 172.21.15.28:0/4047746761 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e0066c20 con 0x7ff9e00c81a0 2024-09-13T16:44:41.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.741+0000 7ff9e4c00640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e00c81a0 0x7ff9e01eebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.741+0000 7ff9e4c00640 1 -- 172.21.15.28:0/4047746761 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9e014e1b0 con 0x7ff9e00c81a0 2024-09-13T16:44:41.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.741+0000 7ff9e4c00640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e00c81a0 0x7ff9e01eebf0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc005880 tx=0x7ff9cc00f9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.742+0000 7ff9dd600640 1 -- 172.21.15.28:0/4047746761 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff9d4004b40 con 0x7ff9e00c17f0 2024-09-13T16:44:41.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.742+0000 7ff9dd600640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff9c006ac30 0x7ff9c006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.742+0000 7ff9dd600640 1 -- 172.21.15.28:0/4047746761 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9d4088b70 con 0x7ff9e00c17f0 2024-09-13T16:44:41.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.743+0000 7ff9e6000640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff9c006ac30 0x7ff9c006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.743+0000 7ff9dd600640 1 -- 172.21.15.28:0/4047746761 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc005ac0 con 0x7ff9e00c81a0 2024-09-13T16:44:41.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.743+0000 7ff9dd600640 1 -- 172.21.15.28:0/4047746761 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7ff9cc005c20 con 0x7ff9e00c81a0 2024-09-13T16:44:41.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.743+0000 7ff9dd600640 1 -- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e00c81a0 msgr2=0x7ff9e01eebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.743+0000 7ff9dd600640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e00c81a0 0x7ff9e01eebf0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc005880 tx=0x7ff9cc00f9d0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.743+0000 7ff9dd600640 1 -- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e00c81a0 msgr2=0x7ff9e01eebf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:44:41.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.743+0000 7ff9dd600640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e00c81a0 0x7ff9e01eebf0 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc005880 tx=0x7ff9cc00f9d0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.743+0000 7ff9e6000640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff9c006ac30 0x7ff9c006d0f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8034000 tx=0x7ff9d802f000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.749+0000 7ff9e6a00640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e0071ac0 0x7ff9e0145690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.749+0000 7ff9e6a00640 1 -- 172.21.15.28:0/4047746761 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e00c2d10 con 0x7ff9e0071ac0 2024-09-13T16:44:41.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.750+0000 7ff9e4c00640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e0071ac0 0x7ff9e0145690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.750+0000 7ff9e4c00640 1 -- 172.21.15.28:0/4047746761 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff9e00c2a30 con 0x7ff9e0071ac0 2024-09-13T16:44:41.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.751+0000 7ff9e4c00640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e0071ac0 0x7ff9e0145690 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc005880 tx=0x7ff9cc0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.751+0000 7ff9dd600640 1 -- 172.21.15.28:0/4047746761 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc023070 con 0x7ff9e0071ac0 2024-09-13T16:44:41.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.751+0000 7ff9dd600640 1 -- 172.21.15.28:0/4047746761 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4789 (secure 0 0 0) 0x7ff9cc0073a0 con 0x7ff9e0071ac0 2024-09-13T16:44:41.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.751+0000 7ff9dd600640 1 -- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e0071ac0 msgr2=0x7ff9e0145690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.751+0000 7ff9dd600640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e0071ac0 0x7ff9e0145690 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc005880 tx=0x7ff9cc0057f0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.751+0000 7ff9dd600640 1 -- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e0071ac0 msgr2=0x7ff9e0145690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:44:41.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.751+0000 7ff9dd600640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9e0071ac0 0x7ff9e0145690 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc005880 tx=0x7ff9cc0057f0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.753 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:44:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:44:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:44:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:44:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 4, 2024-09-13T16:44:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:44:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:44:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:44:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:44:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 62, 2024-09-13T16:44:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:44:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:44:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:44:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:44:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:44:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:44:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:44:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:44:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:44:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:44:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:44:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:44:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:44:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:44:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:44:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:44:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:44:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:44:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:44:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:44:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:44:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:44:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:44:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:44:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:44:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:44:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:44:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:44:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:44:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:44:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:44:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:44:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:44:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:44:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:44:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:44:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:44:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:44:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:44:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:44:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:44:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:44:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:44:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:44:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:44:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:44:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:44:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:44:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:44:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:44:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:44:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:44:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:44:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:44:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:44:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:44:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:44:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:44:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:44:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:44:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:44:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:44:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:44:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:44:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:44:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:44:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:44:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:44:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:44:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:44:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:44:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:44:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:44:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:44:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:44:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:44:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:44:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:44:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:44:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:44:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:44:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:44:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:44:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:44:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:44:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:44:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:44:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:44:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:44:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:44:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:44:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:44:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:44:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:44:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:44:41.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.765 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:41.765 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:41.765 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:44:41.765 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.765 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.765 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:44:41.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.765 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:44:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:44:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:44:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:44:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.767 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:41.767 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:41.767 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:44:41.767 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.767 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.767 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:44:41.767 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:44:41.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.752+0000 7ff9e6a00640 1 -- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff9c006ac30 msgr2=0x7ff9c006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.752+0000 7ff9e6a00640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff9c006ac30 0x7ff9c006d0f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8034000 tx=0x7ff9d802f000 comp rx=0 tx=0).stop 2024-09-13T16:44:41.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.752+0000 7ff9e6a00640 1 -- 172.21.15.28:0/4047746761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e00c17f0 msgr2=0x7ff9e01ef230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.752+0000 7ff9e6a00640 1 --2- 172.21.15.28:0/4047746761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e00c17f0 0x7ff9e01ef230 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400c3e0 tx=0x7ff9d400c8a0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.752+0000 7ff9e6000640 1 -- 172.21.15.28:0/4047746761 reap_dead start 2024-09-13T16:44:41.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.752+0000 7ff9e6000640 1 -- 172.21.15.28:0/4047746761 reap_dead start 2024-09-13T16:44:41.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.752+0000 7ff9e6a00640 1 -- 172.21.15.28:0/4047746761 shutdown_connections 2024-09-13T16:44:41.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.752+0000 7ff9e6a00640 1 -- 172.21.15.28:0/4047746761 >> 172.21.15.28:0/4047746761 conn(0x7ff9e00ca2f0 msgr2=0x7ff9e00c6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:41.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.752+0000 7ff9e6a00640 1 -- 172.21.15.28:0/4047746761 shutdown_connections 2024-09-13T16:44:41.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.752+0000 7ff9e6a00640 1 -- 172.21.15.28:0/4047746761 wait complete. 2024-09-13T16:44:41.770 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:44:41.770 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-13T16:44:41.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T16:44:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.924+0000 7fdd0ac00640 1 Processor -- start 2024-09-13T16:44:41.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.924+0000 7fdd0ac00640 1 -- start start 2024-09-13T16:44:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.925+0000 7fdd0ac00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdd040c73e0 0x7fdd040c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.925+0000 7fdd0ac00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdd040c5a60 0x7fdd0414f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.925+0000 7fdd0ac00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd040c6f50 con 0x7fdd040c5140 2024-09-13T16:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.925+0000 7fdd0ac00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd040c6940 con 0x7fdd040c5a60 2024-09-13T16:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.925+0000 7fdd0ac00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd040c1b50 con 0x7fdd040c73e0 2024-09-13T16:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.925+0000 7fdd03e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdd040c5140 0x7fdd04159800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57226/0 (socket says 172.21.15.28:57226) 2024-09-13T16:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.925+0000 7fdd08c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdd040c5a60 0x7fdd0414f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.925+0000 7fdd03e00640 1 -- 172.21.15.28:0/558408303 learned_addr learned my addr 172.21.15.28:0/558408303 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.926+0000 7fdcfbe00640 1 --2- 172.21.15.28:0/558408303 >> v2:172.21.15.191:3300/0 conn(0x7fdd040c73e0 0x7fdd040c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.926+0000 7fdd08c00640 1 -- 172.21.15.28:0/558408303 >> v2:172.21.15.191:3300/0 conn(0x7fdd040c73e0 msgr2=0x7fdd040c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.926+0000 7fdd08c00640 1 --2- 172.21.15.28:0/558408303 >> v2:172.21.15.191:3300/0 conn(0x7fdd040c73e0 0x7fdd040c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.926+0000 7fdd08c00640 1 -- 172.21.15.28:0/558408303 >> v1:172.21.15.28:6789/0 conn(0x7fdd040c5140 legacy=0x7fdd04159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.926+0000 7fdd08c00640 1 -- 172.21.15.28:0/558408303 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd040c1e30 con 0x7fdd040c5a60 2024-09-13T16:44:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.927+0000 7fdd08c00640 1 --2- 172.21.15.28:0/558408303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd040c5a60 0x7fdd0414f540 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdd0406aed0 tx=0x7fdcfc02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e51a74f8c650b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.927+0000 7fdd03400640 1 -- 172.21.15.28:0/558408303 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc007740 con 0x7fdd040c5a60 2024-09-13T16:44:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.927+0000 7fdd03400640 1 -- 172.21.15.28:0/558408303 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcfc02d050 con 0x7fdd040c5a60 2024-09-13T16:44:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.927+0000 7fdd03400640 1 -- 172.21.15.28:0/558408303 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc007ac0 con 0x7fdd040c5a60 2024-09-13T16:44:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.928+0000 7fdd0ac00640 1 -- 172.21.15.28:0/558408303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd040c5a60 msgr2=0x7fdd0414f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.928+0000 7fdd0ac00640 1 --2- 172.21.15.28:0/558408303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd040c5a60 0x7fdd0414f540 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdd0406aed0 tx=0x7fdcfc02a5f0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.928+0000 7fdd0ac00640 1 -- 172.21.15.28:0/558408303 shutdown_connections 2024-09-13T16:44:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.928+0000 7fdd0ac00640 1 --2- 172.21.15.28:0/558408303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd040c5a60 0x7fdd0414f540 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.928+0000 7fdd0ac00640 1 --2- 172.21.15.28:0/558408303 >> v2:172.21.15.191:3300/0 conn(0x7fdd040c73e0 0x7fdd040c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.928+0000 7fdd0ac00640 1 -- 172.21.15.28:0/558408303 >> 172.21.15.28:0/558408303 conn(0x7fdd040ca6b0 msgr2=0x7fdd040caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.928+0000 7fdd0ac00640 1 -- 172.21.15.28:0/558408303 shutdown_connections 2024-09-13T16:44:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.928+0000 7fdd0ac00640 1 -- 172.21.15.28:0/558408303 wait complete. 2024-09-13T16:44:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.930+0000 7fdd0ac00640 1 Processor -- start 2024-09-13T16:44:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.930+0000 7fdd0ac00640 1 -- start start 2024-09-13T16:44:41.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.930+0000 7fdd0ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd040c5140 0x7fdd04141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.931+0000 7fdd0ac00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd040c5a60 0x7fdd041f0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.931+0000 7fdd0ac00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd040c73e0 0x7fdd041f0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.931+0000 7fdd0ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd041508e0 con 0x7fdd040c5140 2024-09-13T16:44:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.931+0000 7fdd0ac00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd04159320 con 0x7fdd040c73e0 2024-09-13T16:44:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.931+0000 7fdd0ac00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd04141650 con 0x7fdd040c5a60 2024-09-13T16:44:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.931+0000 7fdd03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd040c5140 0x7fdd04141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.931+0000 7fdcfbe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd040c5a60 0x7fdd041f0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.931+0000 7fdcfbe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd040c5a60 0x7fdd041f0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:41136/0 (socket says 172.21.15.28:41136) 2024-09-13T16:44:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.931+0000 7fdcfbe00640 1 -- 172.21.15.28:0/3175239693 learned_addr learned my addr 172.21.15.28:0/3175239693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.931+0000 7fdd08c00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd040c73e0 0x7fdd041f0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.931+0000 7fdd03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd040c5140 0x7fdd04141110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60362/0 (socket says 172.21.15.28:60362) 2024-09-13T16:44:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.931+0000 7fdcfbe00640 1 -- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd040c73e0 msgr2=0x7fdd041f0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.931+0000 7fdcfbe00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd040c73e0 0x7fdd041f0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.931+0000 7fdcfbe00640 1 -- 172.21.15.28:0/3175239693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd040c5140 msgr2=0x7fdd04141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.931+0000 7fdcfbe00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd040c5140 0x7fdd04141110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.931+0000 7fdcfbe00640 1 -- 172.21.15.28:0/3175239693 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc009a70 con 0x7fdd040c5a60 2024-09-13T16:44:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.932+0000 7fdcfbe00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd040c5a60 0x7fdd041f0680 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdcec00e9b0 tx=0x7fdcec00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.932+0000 7fdd00c00640 1 -- 172.21.15.28:0/3175239693 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcec009670 con 0x7fdd040c5a60 2024-09-13T16:44:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.932+0000 7fdd00c00640 1 -- 172.21.15.28:0/3175239693 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcec00c9a0 con 0x7fdd040c5a60 2024-09-13T16:44:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.932+0000 7fdd00c00640 1 -- 172.21.15.28:0/3175239693 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcec00cd50 con 0x7fdd040c5a60 2024-09-13T16:44:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.932+0000 7fdd0ac00640 1 -- 172.21.15.28:0/3175239693 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd04158d60 con 0x7fdd040c5a60 2024-09-13T16:44:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.932+0000 7fdd0ac00640 1 -- 172.21.15.28:0/3175239693 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd041efba0 con 0x7fdd040c5a60 2024-09-13T16:44:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.934+0000 7fdd0ac00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd041efdc0 0x7fdd041f1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.934+0000 7fdd0ac00640 1 -- 172.21.15.28:0/3175239693 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd04066c20 con 0x7fdd041efdc0 2024-09-13T16:44:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.934+0000 7fdd03e00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd041efdc0 0x7fdd041f1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.934+0000 7fdd00c00640 1 -- 172.21.15.28:0/3175239693 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdcec013470 con 0x7fdd040c5a60 2024-09-13T16:44:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.934+0000 7fdd03e00640 1 -- 172.21.15.28:0/3175239693 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd0406fe80 con 0x7fdd041efdc0 2024-09-13T16:44:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.935+0000 7fdd03e00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd041efdc0 0x7fdd041f1b70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4000f60 tx=0x7fdcf4001880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.935+0000 7fdd00c00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fdce006ac30 0x7fdce006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.935+0000 7fdd00c00640 1 -- 172.21.15.28:0/3175239693 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdcec058440 con 0x7fdd040c5a60 2024-09-13T16:44:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.935+0000 7fdd08c00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fdce006ac30 0x7fdce006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.936+0000 7fdd00c00640 1 -- 172.21.15.28:0/3175239693 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4017070 con 0x7fdd041efdc0 2024-09-13T16:44:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.936+0000 7fdd00c00640 1 -- 172.21.15.28:0/3175239693 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fdcf4011580 con 0x7fdd041efdc0 2024-09-13T16:44:41.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.936+0000 7fdd00c00640 1 -- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd041efdc0 msgr2=0x7fdd041f1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.936+0000 7fdd00c00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd041efdc0 0x7fdd041f1b70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4000f60 tx=0x7fdcf4001880 comp rx=0 tx=0).stop 2024-09-13T16:44:41.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.936+0000 7fdd00c00640 1 -- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd041efdc0 msgr2=0x7fdd041f1b70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:44:41.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.936+0000 7fdd00c00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd041efdc0 0x7fdd041f1b70 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4000f60 tx=0x7fdcf4001880 comp rx=0 tx=0).stop 2024-09-13T16:44:41.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.936+0000 7fdd08c00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fdce006ac30 0x7fdce006d0f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc034000 tx=0x7fdcfc02f000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.942+0000 7fdd0ac00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd041f20b0 0x7fdd041f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:41.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.942+0000 7fdd0ac00640 1 -- 172.21.15.28:0/3175239693 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd040c3750 con 0x7fdd041f20b0 2024-09-13T16:44:41.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.943+0000 7fdd03e00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd041f20b0 0x7fdd041f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:41.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.943+0000 7fdd03e00640 1 -- 172.21.15.28:0/3175239693 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdd04149830 con 0x7fdd041f20b0 2024-09-13T16:44:41.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.943+0000 7fdd03e00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd041f20b0 0x7fdd041f24c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4000f60 tx=0x7fdcf4000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:41.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.944+0000 7fdd00c00640 1 -- 172.21.15.28:0/3175239693 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4023070 con 0x7fdd041f20b0 2024-09-13T16:44:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.944+0000 7fdd00c00640 1 -- 172.21.15.28:0/3175239693 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4955 (secure 0 0 0) 0x7fdcf4011cc0 con 0x7fdd041f20b0 2024-09-13T16:44:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.944+0000 7fdd00c00640 1 -- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd041f20b0 msgr2=0x7fdd041f24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.944+0000 7fdd00c00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd041f20b0 0x7fdd041f24c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4000f60 tx=0x7fdcf4000ed0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.944+0000 7fdd00c00640 1 -- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd041f20b0 msgr2=0x7fdd041f24c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:44:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.944+0000 7fdd00c00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd041f20b0 0x7fdd041f24c0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4000f60 tx=0x7fdcf4000ed0 comp rx=0 tx=0).stop 2024-09-13T16:44:41.946 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:44:41.946 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:44:41.946 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:44:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:44:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 4, 2024-09-13T16:44:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:44:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:44:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:44:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:44:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 62, 2024-09-13T16:44:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:44:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:44:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:44:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:44:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:44:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:44:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:44:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:44:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:44:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:44:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:44:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:44:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:44:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:44:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:44:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:44:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:44:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:44:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:44:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:44:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:44:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:44:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:44:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:44:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:44:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:44:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:44:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:44:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:44:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:44:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:44:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:44:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:44:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:44:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:44:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:44:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:44:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:44:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:44:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:44:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:44:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:44:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:44:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:44:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:44:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:44:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:44:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:44:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:44:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:44:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:44:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:44:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:44:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:44:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:44:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:44:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:44:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:44:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:44:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:44:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:44:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:44:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:44:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:44:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:44:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:44:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:44:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:44:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:44:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:44:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:44:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:44:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:44:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:44:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:44:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:44:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:44:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:44:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:44:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:44:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:44:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:44:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:44:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:44:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:44:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:44:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:44:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:44:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:44:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:44:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:44:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:44:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:44:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:44:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:44:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:44:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:44:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4 2024-09-13T16:44:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:44:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4 2024-09-13T16:44:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:44:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 5 2024-09-13T16:44:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": [ 2024-09-13T16:44:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:44:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:44:41.959 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.945+0000 7fdd0ac00640 1 -- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fdce006ac30 msgr2=0x7fdce006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.945+0000 7fdd0ac00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fdce006ac30 0x7fdce006d0f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc034000 tx=0x7fdcfc02f000 comp rx=0 tx=0).stop 2024-09-13T16:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.945+0000 7fdd0ac00640 1 -- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd040c5a60 msgr2=0x7fdd041f0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.945+0000 7fdd0ac00640 1 --2- 172.21.15.28:0/3175239693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd040c5a60 0x7fdd041f0680 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdcec00e9b0 tx=0x7fdcec00ee70 comp rx=0 tx=0).stop 2024-09-13T16:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.945+0000 7fdd08c00640 1 -- 172.21.15.28:0/3175239693 reap_dead start 2024-09-13T16:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.945+0000 7fdd08c00640 1 -- 172.21.15.28:0/3175239693 reap_dead start 2024-09-13T16:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.945+0000 7fdd0ac00640 1 -- 172.21.15.28:0/3175239693 shutdown_connections 2024-09-13T16:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.945+0000 7fdd0ac00640 1 -- 172.21.15.28:0/3175239693 >> 172.21.15.28:0/3175239693 conn(0x7fdd040ca6b0 msgr2=0x7fdd040c9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.945+0000 7fdd0ac00640 1 -- 172.21.15.28:0/3175239693 shutdown_connections 2024-09-13T16:44:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:41.945+0000 7fdd0ac00640 1 -- 172.21.15.28:0/3175239693 wait complete. 2024-09-13T16:44:41.962 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:44:41.962 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-13T16:44:41.962 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-13T16:44:41.962 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-13T16:44:41.962 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-13T16:44:41.963 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-13T16:44:41.963 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:44:41.979 INFO:tasks.ceph.mon.c:Stopped 2024-09-13T16:44:41.979 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-13T16:44:41.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:44:42.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.131+0000 7fe1e7e00640 1 Processor -- start 2024-09-13T16:44:42.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.132+0000 7fe1e7e00640 1 -- start start 2024-09-13T16:44:42.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.133+0000 7fe1e7e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e80c5060 0x7fe1e80c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:42.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.133+0000 7fe1e7e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe1e8156610 0x7fe1e8150400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:42.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.133+0000 7fe1e7e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e80c5df0 con 0x7fe1e80c5440 2024-09-13T16:44:42.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.133+0000 7fe1e7e00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1e80c2ab0 con 0x7fe1e8156610 2024-09-13T16:44:42.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.133+0000 7fe1e7e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e80c2d70 con 0x7fe1e80c5060 2024-09-13T16:44:42.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.133+0000 7fe1e6a00640 1 -- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e80c5060 msgr2=0x7fe1e80c44d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3301/0 2024-09-13T16:44:42.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.133+0000 7fe1dfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe1e80c5440 0x7fe1e80c2300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57242/0 (socket says 172.21.15.28:57242) 2024-09-13T16:44:42.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.133+0000 7fe1dfe00640 1 -- 172.21.15.28:0/3578716712 learned_addr learned my addr 172.21.15.28:0/3578716712 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:42.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.133+0000 7fe1e6a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e80c5060 0x7fe1e80c44d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:44:42.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.133+0000 7fe1e7400640 1 --2- 172.21.15.28:0/3578716712 >> v2:172.21.15.191:3300/0 conn(0x7fe1e8156610 0x7fe1e8150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:42.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.133+0000 7fe1e6000640 1 -- 172.21.15.28:0/3578716712 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210981347 0 0) 0x7fe1e80c5df0 con 0x7fe1e80c5440 2024-09-13T16:44:42.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.133+0000 7fe1e6000640 1 -- 172.21.15.28:0/3578716712 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1cc003520 con 0x7fe1e80c5440 2024-09-13T16:44:42.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.134+0000 7fe1e7400640 1 -- 172.21.15.28:0/3578716712 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e80c5060 msgr2=0x7fe1e80c44d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:44:42.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.134+0000 7fe1e7400640 1 --2- 172.21.15.28:0/3578716712 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e80c5060 0x7fe1e80c44d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:42.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.134+0000 7fe1e7400640 1 -- 172.21.15.28:0/3578716712 >> v1:172.21.15.28:6789/0 conn(0x7fe1e80c5440 legacy=0x7fe1e80c2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:42.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.134+0000 7fe1e7400640 1 -- 172.21.15.28:0/3578716712 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e80c6040 con 0x7fe1e8156610 2024-09-13T16:44:42.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.135+0000 7fe1e7400640 1 --2- 172.21.15.28:0/3578716712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1e8156610 0x7fe1e8150400 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0009d90 tx=0x7fe1e002a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f64747a5aa4876be server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:42.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.135+0000 7fe1e6000640 1 -- 172.21.15.28:0/3578716712 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0007740 con 0x7fe1e8156610 2024-09-13T16:44:42.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.135+0000 7fe1e6000640 1 -- 172.21.15.28:0/3578716712 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e002d050 con 0x7fe1e8156610 2024-09-13T16:44:42.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.135+0000 7fe1e6000640 1 -- 172.21.15.28:0/3578716712 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0007ac0 con 0x7fe1e8156610 2024-09-13T16:44:42.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.135+0000 7fe1e7e00640 1 -- 172.21.15.28:0/3578716712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1e8156610 msgr2=0x7fe1e8150400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:42.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.135+0000 7fe1e7e00640 1 --2- 172.21.15.28:0/3578716712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1e8156610 0x7fe1e8150400 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0009d90 tx=0x7fe1e002a710 comp rx=0 tx=0).stop 2024-09-13T16:44:42.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.135+0000 7fe1e7e00640 1 -- 172.21.15.28:0/3578716712 shutdown_connections 2024-09-13T16:44:42.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.135+0000 7fe1e7e00640 1 --2- 172.21.15.28:0/3578716712 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1e8156610 0x7fe1e8150400 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:42.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.135+0000 7fe1e7e00640 1 --2- 172.21.15.28:0/3578716712 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e80c5060 0x7fe1e80c44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:42.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.135+0000 7fe1e7e00640 1 -- 172.21.15.28:0/3578716712 >> 172.21.15.28:0/3578716712 conn(0x7fe1e8069e80 msgr2=0x7fe1e806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:42.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.136+0000 7fe1e7e00640 1 -- 172.21.15.28:0/3578716712 shutdown_connections 2024-09-13T16:44:42.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.136+0000 7fe1e7e00640 1 -- 172.21.15.28:0/3578716712 wait complete. 2024-09-13T16:44:42.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.137+0000 7fe1e7e00640 1 Processor -- start 2024-09-13T16:44:42.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.137+0000 7fe1e7e00640 1 -- start start 2024-09-13T16:44:42.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.138+0000 7fe1e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e80c5060 0x7fe1e8149a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:42.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.138+0000 7fe1e7e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1e80c5440 0x7fe1e8149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:42.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.138+0000 7fe1e7e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e8156610 0x7fe1e80ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:42.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.138+0000 7fe1e7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e8151610 con 0x7fe1e80c5060 2024-09-13T16:44:42.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.138+0000 7fe1e7e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e81558b0 con 0x7fe1e80c5440 2024-09-13T16:44:42.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.138+0000 7fe1e7e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e80c91f0 con 0x7fe1e8156610 2024-09-13T16:44:42.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.138+0000 7fe1e7400640 1 -- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e8156610 msgr2=0x7fe1e80ca750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.191:3301/0 2024-09-13T16:44:42.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.138+0000 7fe1e6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e80c5060 0x7fe1e8149a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:42.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.138+0000 7fe1e7400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e8156610 0x7fe1e80ca750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:44:42.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.138+0000 7fe1e6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e80c5060 0x7fe1e8149a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60434/0 (socket says 172.21.15.28:60434) 2024-09-13T16:44:42.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.138+0000 7fe1e6a00640 1 -- 172.21.15.28:0/2513989009 learned_addr learned my addr 172.21.15.28:0/2513989009 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:42.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.139+0000 7fe1dfe00640 1 --2- 172.21.15.28:0/2513989009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1e80c5440 0x7fe1e8149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:42.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.139+0000 7fe1e6a00640 1 -- 172.21.15.28:0/2513989009 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e8156610 msgr2=0x7fe1e80ca750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:44:42.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.139+0000 7fe1e6a00640 1 --2- 172.21.15.28:0/2513989009 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e8156610 0x7fe1e80ca750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:42.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.139+0000 7fe1e6a00640 1 -- 172.21.15.28:0/2513989009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1e80c5440 msgr2=0x7fe1e8149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:42.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.139+0000 7fe1e6a00640 1 --2- 172.21.15.28:0/2513989009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1e80c5440 0x7fe1e8149f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:42.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.139+0000 7fe1e6a00640 1 -- 172.21.15.28:0/2513989009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e0009a70 con 0x7fe1e80c5060 2024-09-13T16:44:42.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.139+0000 7fe1e6a00640 1 --2- 172.21.15.28:0/2513989009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e80c5060 0x7fe1e8149a00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe1d800b5b0 tx=0x7fe1d800ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:42.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.139+0000 7fe1dea00640 1 -- 172.21.15.28:0/2513989009 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d8003dc0 con 0x7fe1e80c5060 2024-09-13T16:44:42.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.139+0000 7fe1dea00640 1 -- 172.21.15.28:0/2513989009 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d8005070 con 0x7fe1e80c5060 2024-09-13T16:44:42.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.139+0000 7fe1dea00640 1 -- 172.21.15.28:0/2513989009 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d8005420 con 0x7fe1e80c5060 2024-09-13T16:44:42.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.139+0000 7fe1e7e00640 1 -- 172.21.15.28:0/2513989009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e80c8c70 con 0x7fe1e80c5060 2024-09-13T16:44:42.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.140+0000 7fe1e7e00640 1 -- 172.21.15.28:0/2513989009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e8150f50 con 0x7fe1e80c5060 2024-09-13T16:44:42.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.140+0000 7fe1e7e00640 1 -- 172.21.15.28:0/2513989009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e80c7d00 con 0x7fe1e80c5060 2024-09-13T16:44:42.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.141+0000 7fe1dea00640 1 -- 172.21.15.28:0/2513989009 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe1d8005580 con 0x7fe1e80c5060 2024-09-13T16:44:42.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.142+0000 7fe1dea00640 1 --2- 172.21.15.28:0/2513989009 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe1c406abb0 0x7fe1c406d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:42.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.142+0000 7fe1dea00640 1 -- 172.21.15.28:0/2513989009 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1d8089340 con 0x7fe1e80c5060 2024-09-13T16:44:42.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.142+0000 7fe1dfe00640 1 --2- 172.21.15.28:0/2513989009 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe1c406abb0 0x7fe1c406d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:42.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.143+0000 7fe1dfe00640 1 --2- 172.21.15.28:0/2513989009 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe1c406abb0 0x7fe1c406d070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe1e81e57a0 tx=0x7fe1d0007d30 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.146+0000 7fe1dea00640 1 -- 172.21.15.28:0/2513989009 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe1d80543f0 con 0x7fe1e80c5060 2024-09-13T16:44:42.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.390+0000 7fe1e7e00640 1 -- 172.21.15.28:0/2513989009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe1e80c3cf0 con 0x7fe1e80c5060 2024-09-13T16:44:42.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.391+0000 7fe1dea00640 1 -- 172.21.15.28:0/2513989009 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7fe1d8013620 con 0x7fe1e80c5060 2024-09-13T16:44:42.392 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":62,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:44:42.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.393+0000 7fe1e7e00640 1 -- 172.21.15.28:0/2513989009 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe1c406abb0 msgr2=0x7fe1c406d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:42.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.393+0000 7fe1e7e00640 1 --2- 172.21.15.28:0/2513989009 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe1c406abb0 0x7fe1c406d070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe1e81e57a0 tx=0x7fe1d0007d30 comp rx=0 tx=0).stop 2024-09-13T16:44:42.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.393+0000 7fe1e7e00640 1 -- 172.21.15.28:0/2513989009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e80c5060 msgr2=0x7fe1e8149a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:42.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.393+0000 7fe1e7e00640 1 --2- 172.21.15.28:0/2513989009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e80c5060 0x7fe1e8149a00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe1d800b5b0 tx=0x7fe1d800ba70 comp rx=0 tx=0).stop 2024-09-13T16:44:42.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.393+0000 7fe1e7e00640 1 -- 172.21.15.28:0/2513989009 shutdown_connections 2024-09-13T16:44:42.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.393+0000 7fe1e7e00640 1 --2- 172.21.15.28:0/2513989009 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe1c406abb0 0x7fe1c406d070 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:42.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.393+0000 7fe1e7e00640 1 --2- 172.21.15.28:0/2513989009 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e8156610 0x7fe1e80ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:42.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.393+0000 7fe1e7e00640 1 --2- 172.21.15.28:0/2513989009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1e80c5440 0x7fe1e8149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:42.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.393+0000 7fe1e7e00640 1 --2- 172.21.15.28:0/2513989009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e80c5060 0x7fe1e8149a00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:42.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.393+0000 7fe1e7e00640 1 -- 172.21.15.28:0/2513989009 >> 172.21.15.28:0/2513989009 conn(0x7fe1e8069e80 msgr2=0x7fe1e80c1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:42.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.393+0000 7fe1e7e00640 1 -- 172.21.15.28:0/2513989009 shutdown_connections 2024-09-13T16:44:42.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:42.393+0000 7fe1e7e00640 1 -- 172.21.15.28:0/2513989009 wait complete. 2024-09-13T16:44:45.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:44:45.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.576+0000 7f8e97200640 1 Processor -- start 2024-09-13T16:44:45.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.576+0000 7f8e97200640 1 -- start start 2024-09-13T16:44:45.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.577+0000 7f8e97200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e980c5060 0x7f8e980c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:45.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.577+0000 7f8e97200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e98156610 0x7f8e98150400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:45.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.577+0000 7f8e97200640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e980c5df0 con 0x7f8e980c5440 2024-09-13T16:44:45.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.577+0000 7f8e97200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e980c2ab0 con 0x7f8e98156610 2024-09-13T16:44:45.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.577+0000 7f8e97200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e980c2d70 con 0x7f8e980c5060 2024-09-13T16:44:45.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.577+0000 7f8e95e00640 1 -- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e980c5060 msgr2=0x7f8e980c44d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3301/0 2024-09-13T16:44:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.577+0000 7f8e95e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e980c5060 0x7f8e980c44d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:44:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.577+0000 7f8e96800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e98156610 0x7f8e98150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.577+0000 7f8e8d400640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f8e980c5440 0x7f8e980c2300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:51424/0 (socket says 172.21.15.28:51424) 2024-09-13T16:44:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.577+0000 7f8e8d400640 1 -- 172.21.15.28:0/1926890995 learned_addr learned my addr 172.21.15.28:0/1926890995 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:44:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.578+0000 7f8e96800640 1 -- 172.21.15.28:0/1926890995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e980c5060 msgr2=0x7f8e980c44d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:44:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.578+0000 7f8e96800640 1 --2- 172.21.15.28:0/1926890995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e980c5060 0x7f8e980c44d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.578+0000 7f8e96800640 1 -- 172.21.15.28:0/1926890995 >> v1:172.21.15.191:6789/0 conn(0x7f8e980c5440 legacy=0x7f8e980c2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.578+0000 7f8e96800640 1 -- 172.21.15.28:0/1926890995 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e980c6040 con 0x7f8e98156610 2024-09-13T16:44:45.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.579+0000 7f8e96800640 1 --2- 172.21.15.28:0/1926890995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e98156610 0x7f8e98150400 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8e900097a0 tx=0x7f8e9002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90eee50fa4478530 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:45.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.579+0000 7f8e95400640 1 -- 172.21.15.28:0/1926890995 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90032440 con 0x7f8e98156610 2024-09-13T16:44:45.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.579+0000 7f8e95400640 1 -- 172.21.15.28:0/1926890995 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e900325a0 con 0x7f8e98156610 2024-09-13T16:44:45.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.579+0000 7f8e95400640 1 -- 172.21.15.28:0/1926890995 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90032930 con 0x7f8e98156610 2024-09-13T16:44:45.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.579+0000 7f8e97200640 1 -- 172.21.15.28:0/1926890995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e98156610 msgr2=0x7f8e98150400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.580+0000 7f8e97200640 1 --2- 172.21.15.28:0/1926890995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e98156610 0x7f8e98150400 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8e900097a0 tx=0x7f8e9002a250 comp rx=0 tx=0).stop 2024-09-13T16:44:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.580+0000 7f8e97200640 1 -- 172.21.15.28:0/1926890995 shutdown_connections 2024-09-13T16:44:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.580+0000 7f8e97200640 1 --2- 172.21.15.28:0/1926890995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e98156610 0x7f8e98150400 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.580+0000 7f8e97200640 1 --2- 172.21.15.28:0/1926890995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e980c5060 0x7f8e980c44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.580+0000 7f8e97200640 1 -- 172.21.15.28:0/1926890995 >> 172.21.15.28:0/1926890995 conn(0x7f8e98069e80 msgr2=0x7f8e9806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.580+0000 7f8e97200640 1 -- 172.21.15.28:0/1926890995 shutdown_connections 2024-09-13T16:44:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.580+0000 7f8e97200640 1 -- 172.21.15.28:0/1926890995 wait complete. 2024-09-13T16:44:45.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.581+0000 7f8e97200640 1 Processor -- start 2024-09-13T16:44:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.582+0000 7f8e97200640 1 -- start start 2024-09-13T16:44:45.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.583+0000 7f8e97200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e980c5060 0x7f8e980ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:45.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.583+0000 7f8e97200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e980c5440 0x7f8e9814b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:45.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.583+0000 7f8e97200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e98156610 0x7f8e981e4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:45.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.583+0000 7f8e97200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e981f29e0 con 0x7f8e980c5440 2024-09-13T16:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.583+0000 7f8e97200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e980bb5f0 con 0x7f8e98156610 2024-09-13T16:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.583+0000 7f8e97200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e980ca630 con 0x7f8e980c5060 2024-09-13T16:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.583+0000 7f8e95e00640 1 -- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e980c5060 msgr2=0x7f8e980ca0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.191:3301/0 2024-09-13T16:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.583+0000 7f8e95e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e980c5060 0x7f8e980ca0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.583+0000 7f8e8d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e980c5440 0x7f8e9814b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.583+0000 7f8e96800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e98156610 0x7f8e981e4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.583+0000 7f8e96800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e98156610 0x7f8e981e4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:50832/0 (socket says 172.21.15.28:50832) 2024-09-13T16:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.583+0000 7f8e96800640 1 -- 172.21.15.28:0/3388839204 learned_addr learned my addr 172.21.15.28:0/3388839204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.583+0000 7f8e8d400640 1 -- 172.21.15.28:0/3388839204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e980c5060 msgr2=0x7f8e980ca0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.583+0000 7f8e8d400640 1 --2- 172.21.15.28:0/3388839204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e980c5060 0x7f8e980ca0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.583+0000 7f8e8d400640 1 -- 172.21.15.28:0/3388839204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e98156610 msgr2=0x7f8e981e4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.583+0000 7f8e8d400640 1 --2- 172.21.15.28:0/3388839204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e98156610 0x7f8e981e4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.583+0000 7f8e8d400640 1 -- 172.21.15.28:0/3388839204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e90009400 con 0x7f8e980c5440 2024-09-13T16:44:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.584+0000 7f8e8d400640 1 --2- 172.21.15.28:0/3388839204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e980c5440 0x7f8e9814b9d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000c410 tx=0x7f8e80001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.584+0000 7f8e8ea00640 1 -- 172.21.15.28:0/3388839204 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e80005ce0 con 0x7f8e980c5440 2024-09-13T16:44:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.584+0000 7f8e8ea00640 1 -- 172.21.15.28:0/3388839204 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e80005e40 con 0x7f8e980c5440 2024-09-13T16:44:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.584+0000 7f8e8ea00640 1 -- 172.21.15.28:0/3388839204 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e800072d0 con 0x7f8e980c5440 2024-09-13T16:44:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.584+0000 7f8e97200640 1 -- 172.21.15.28:0/3388839204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e981f2440 con 0x7f8e980c5440 2024-09-13T16:44:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.584+0000 7f8e97200640 1 -- 172.21.15.28:0/3388839204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e981f2700 con 0x7f8e980c5440 2024-09-13T16:44:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.585+0000 7f8e97200640 1 -- 172.21.15.28:0/3388839204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e980c7d00 con 0x7f8e980c5440 2024-09-13T16:44:45.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.586+0000 7f8e8ea00640 1 -- 172.21.15.28:0/3388839204 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8e80007430 con 0x7f8e980c5440 2024-09-13T16:44:45.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.587+0000 7f8e8ea00640 1 --2- 172.21.15.28:0/3388839204 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8e7406abb0 0x7f8e7406d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:45.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.587+0000 7f8e8ea00640 1 -- 172.21.15.28:0/3388839204 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e80088840 con 0x7f8e980c5440 2024-09-13T16:44:45.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.587+0000 7f8e95e00640 1 --2- 172.21.15.28:0/3388839204 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8e7406abb0 0x7f8e7406d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:45.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.588+0000 7f8e95e00640 1 --2- 172.21.15.28:0/3388839204 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8e7406abb0 0x7f8e7406d070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8e88003cb0 tx=0x7f8e88009270 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.592+0000 7f8e8ea00640 1 -- 172.21.15.28:0/3388839204 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8e800539a0 con 0x7f8e980c5440 2024-09-13T16:44:45.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.837+0000 7f8e97200640 1 -- 172.21.15.28:0/3388839204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8e980c9c70 con 0x7f8e980c5440 2024-09-13T16:44:45.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.837+0000 7f8e8ea00640 1 -- 172.21.15.28:0/3388839204 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7f8e980c9c70 con 0x7f8e980c5440 2024-09-13T16:44:45.839 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":66,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:44:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.839+0000 7f8e97200640 1 -- 172.21.15.28:0/3388839204 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8e7406abb0 msgr2=0x7f8e7406d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.839+0000 7f8e97200640 1 --2- 172.21.15.28:0/3388839204 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8e7406abb0 0x7f8e7406d070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8e88003cb0 tx=0x7f8e88009270 comp rx=0 tx=0).stop 2024-09-13T16:44:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.839+0000 7f8e97200640 1 -- 172.21.15.28:0/3388839204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e980c5440 msgr2=0x7f8e9814b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.839+0000 7f8e97200640 1 --2- 172.21.15.28:0/3388839204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e980c5440 0x7f8e9814b9d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000c410 tx=0x7f8e80001e70 comp rx=0 tx=0).stop 2024-09-13T16:44:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.839+0000 7f8e97200640 1 -- 172.21.15.28:0/3388839204 shutdown_connections 2024-09-13T16:44:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.839+0000 7f8e97200640 1 --2- 172.21.15.28:0/3388839204 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8e7406abb0 0x7f8e7406d070 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.839+0000 7f8e97200640 1 --2- 172.21.15.28:0/3388839204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e98156610 0x7f8e981e4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.839+0000 7f8e97200640 1 --2- 172.21.15.28:0/3388839204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e980c5440 0x7f8e9814b9d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.839+0000 7f8e97200640 1 --2- 172.21.15.28:0/3388839204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e980c5060 0x7f8e980ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.839+0000 7f8e97200640 1 -- 172.21.15.28:0/3388839204 >> 172.21.15.28:0/3388839204 conn(0x7f8e98069e80 msgr2=0x7f8e98154d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.840+0000 7f8e97200640 1 -- 172.21.15.28:0/3388839204 shutdown_connections 2024-09-13T16:44:45.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:45.840+0000 7f8e97200640 1 -- 172.21.15.28:0/3388839204 wait complete. 2024-09-13T16:44:48.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:44:49.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.021+0000 7f0ed9200640 1 Processor -- start 2024-09-13T16:44:49.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.021+0000 7f0ed9200640 1 -- start start 2024-09-13T16:44:49.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.022+0000 7f0ed9200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ed4150a70 0x7f0ed40c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:49.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.022+0000 7f0ed9200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ed40c6720 0x7f0ed40c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:49.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.022+0000 7f0ed9200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed4159f70 con 0x7f0ed40c53a0 2024-09-13T16:44:49.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.022+0000 7f0ed9200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed40c64f0 con 0x7f0ed4150a70 2024-09-13T16:44:49.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.022+0000 7f0ed9200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed40c1ea0 con 0x7f0ed40c6720 2024-09-13T16:44:49.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.023+0000 7f0ed1400640 1 -- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ed40c6720 msgr2=0x7f0ed40c4e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3301/0 2024-09-13T16:44:49.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.023+0000 7f0ed1400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ed40c6720 0x7f0ed40c4e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:44:49.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.023+0000 7f0ed1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ed4150a70 0x7f0ed40c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:49.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.023+0000 7f0ed1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ed4150a70 0x7f0ed40c1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38766/0 (socket says 172.21.15.28:38766) 2024-09-13T16:44:49.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.023+0000 7f0ed1e00640 1 -- 172.21.15.28:0/1973503484 learned_addr learned my addr 172.21.15.28:0/1973503484 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:49.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.023+0000 7f0ed1e00640 1 -- 172.21.15.28:0/1973503484 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ed40c6720 msgr2=0x7f0ed40c4e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:44:49.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.023+0000 7f0ed1e00640 1 --2- 172.21.15.28:0/1973503484 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ed40c6720 0x7f0ed40c4e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:49.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.023+0000 7f0ed1e00640 1 -- 172.21.15.28:0/1973503484 >> v1:172.21.15.28:6789/0 conn(0x7f0ed40c53a0 legacy=0x7f0ed4156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:49.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.023+0000 7f0ed1e00640 1 -- 172.21.15.28:0/1973503484 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed415a1e0 con 0x7f0ed4150a70 2024-09-13T16:44:49.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.024+0000 7f0ed1e00640 1 --2- 172.21.15.28:0/1973503484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed4150a70 0x7f0ed40c1960 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4009870 tx=0x7f0ec402a320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f293c585bcd7f2b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:49.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.024+0000 7f0ed0a00640 1 -- 172.21.15.28:0/1973503484 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4032640 con 0x7f0ed4150a70 2024-09-13T16:44:49.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.024+0000 7f0ed0a00640 1 -- 172.21.15.28:0/1973503484 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec40327a0 con 0x7f0ed4150a70 2024-09-13T16:44:49.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.024+0000 7f0ed0a00640 1 -- 172.21.15.28:0/1973503484 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4032b30 con 0x7f0ed4150a70 2024-09-13T16:44:49.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.025+0000 7f0ed9200640 1 -- 172.21.15.28:0/1973503484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed4150a70 msgr2=0x7f0ed40c1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:49.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.025+0000 7f0ed9200640 1 --2- 172.21.15.28:0/1973503484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed4150a70 0x7f0ed40c1960 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4009870 tx=0x7f0ec402a320 comp rx=0 tx=0).stop 2024-09-13T16:44:49.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.025+0000 7f0ed9200640 1 -- 172.21.15.28:0/1973503484 shutdown_connections 2024-09-13T16:44:49.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.025+0000 7f0ed9200640 1 --2- 172.21.15.28:0/1973503484 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ed40c6720 0x7f0ed40c4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:49.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.025+0000 7f0ed9200640 1 --2- 172.21.15.28:0/1973503484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed4150a70 0x7f0ed40c1960 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:49.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.025+0000 7f0ed9200640 1 -- 172.21.15.28:0/1973503484 >> 172.21.15.28:0/1973503484 conn(0x7f0ed4069e80 msgr2=0x7f0ed406a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:49.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.025+0000 7f0ed9200640 1 -- 172.21.15.28:0/1973503484 shutdown_connections 2024-09-13T16:44:49.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.025+0000 7f0ed9200640 1 -- 172.21.15.28:0/1973503484 wait complete. 2024-09-13T16:44:49.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.027+0000 7f0ed9200640 1 Processor -- start 2024-09-13T16:44:49.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.027+0000 7f0ed9200640 1 -- start start 2024-09-13T16:44:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.028+0000 7f0ed9200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ed40c53a0 0x7f0ed4155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.028+0000 7f0ed9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed40c6720 0x7f0ed41e0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.028+0000 7f0ed9200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ed4150a70 0x7f0ed41ec4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:49.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.028+0000 7f0ed9200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed41599b0 con 0x7f0ed40c6720 2024-09-13T16:44:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.028+0000 7f0ed9200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed4159c90 con 0x7f0ed4150a70 2024-09-13T16:44:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.028+0000 7f0ed9200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed41e1dd0 con 0x7f0ed40c53a0 2024-09-13T16:44:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.028+0000 7f0ed1e00640 1 -- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ed40c53a0 msgr2=0x7f0ed4155a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.191:3301/0 2024-09-13T16:44:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.028+0000 7f0ed1e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ed40c53a0 0x7f0ed4155a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:44:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.028+0000 7f0ed1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed40c6720 0x7f0ed41e0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.028+0000 7f0ed1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed40c6720 0x7f0ed41e0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38770/0 (socket says 172.21.15.28:38770) 2024-09-13T16:44:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.028+0000 7f0ed1400640 1 -- 172.21.15.28:0/3321643471 learned_addr learned my addr 172.21.15.28:0/3321643471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.028+0000 7f0ed2800640 1 --2- 172.21.15.28:0/3321643471 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ed4150a70 0x7f0ed41ec4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.028+0000 7f0ed1400640 1 -- 172.21.15.28:0/3321643471 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ed40c53a0 msgr2=0x7f0ed4155a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:44:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.028+0000 7f0ed1400640 1 --2- 172.21.15.28:0/3321643471 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ed40c53a0 0x7f0ed4155a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.028+0000 7f0ed1400640 1 -- 172.21.15.28:0/3321643471 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ed4150a70 msgr2=0x7f0ed41ec4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.028+0000 7f0ed1400640 1 --2- 172.21.15.28:0/3321643471 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ed4150a70 0x7f0ed41ec4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.028+0000 7f0ed1400640 1 -- 172.21.15.28:0/3321643471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec40094d0 con 0x7f0ed40c6720 2024-09-13T16:44:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.029+0000 7f0ed1400640 1 --2- 172.21.15.28:0/3321643471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed40c6720 0x7f0ed41e0fe0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000b550 tx=0x7f0ec000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.029+0000 7f0eca000640 1 -- 172.21.15.28:0/3321643471 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0003dc0 con 0x7f0ed40c6720 2024-09-13T16:44:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.029+0000 7f0eca000640 1 -- 172.21.15.28:0/3321643471 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec0005070 con 0x7f0ed40c6720 2024-09-13T16:44:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.029+0000 7f0eca000640 1 -- 172.21.15.28:0/3321643471 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0005420 con 0x7f0ed40c6720 2024-09-13T16:44:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.029+0000 7f0ed9200640 1 -- 172.21.15.28:0/3321643471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed41e20b0 con 0x7f0ed40c6720 2024-09-13T16:44:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.029+0000 7f0ed9200640 1 -- 172.21.15.28:0/3321643471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed41e2670 con 0x7f0ed40c6720 2024-09-13T16:44:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.030+0000 7f0ed9200640 1 -- 172.21.15.28:0/3321643471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed40c4750 con 0x7f0ed40c6720 2024-09-13T16:44:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.031+0000 7f0eca000640 1 -- 172.21.15.28:0/3321643471 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0ec0005580 con 0x7f0ed40c6720 2024-09-13T16:44:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.032+0000 7f0eca000640 1 --2- 172.21.15.28:0/3321643471 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0eb406abb0 0x7f0eb406d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.032+0000 7f0eca000640 1 -- 172.21.15.28:0/3321643471 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ec0089290 con 0x7f0ed40c6720 2024-09-13T16:44:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.032+0000 7f0ed1e00640 1 --2- 172.21.15.28:0/3321643471 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0eb406abb0 0x7f0eb406d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.033+0000 7f0ed1e00640 1 --2- 172.21.15.28:0/3321643471 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0eb406abb0 0x7f0eb406d070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4034000 tx=0x7f0ec402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.036+0000 7f0eca000640 1 -- 172.21.15.28:0/3321643471 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0ec00543d0 con 0x7f0ed40c6720 2024-09-13T16:44:49.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.280+0000 7f0ed9200640 1 -- 172.21.15.28:0/3321643471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0ed40c5930 con 0x7f0ed40c6720 2024-09-13T16:44:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.280+0000 7f0eca000640 1 -- 172.21.15.28:0/3321643471 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7f0ec00579f0 con 0x7f0ed40c6720 2024-09-13T16:44:49.282 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":69,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:44:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.282+0000 7f0ed9200640 1 -- 172.21.15.28:0/3321643471 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0eb406abb0 msgr2=0x7f0eb406d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.282+0000 7f0ed9200640 1 --2- 172.21.15.28:0/3321643471 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0eb406abb0 0x7f0eb406d070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4034000 tx=0x7f0ec402e000 comp rx=0 tx=0).stop 2024-09-13T16:44:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.282+0000 7f0ed9200640 1 -- 172.21.15.28:0/3321643471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed40c6720 msgr2=0x7f0ed41e0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.282+0000 7f0ed9200640 1 --2- 172.21.15.28:0/3321643471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed40c6720 0x7f0ed41e0fe0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000b550 tx=0x7f0ec000ba10 comp rx=0 tx=0).stop 2024-09-13T16:44:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.282+0000 7f0ed9200640 1 -- 172.21.15.28:0/3321643471 shutdown_connections 2024-09-13T16:44:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.282+0000 7f0ed9200640 1 --2- 172.21.15.28:0/3321643471 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0eb406abb0 0x7f0eb406d070 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.282+0000 7f0ed9200640 1 --2- 172.21.15.28:0/3321643471 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ed4150a70 0x7f0ed41ec4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.282+0000 7f0ed9200640 1 --2- 172.21.15.28:0/3321643471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ed40c6720 0x7f0ed41e0fe0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.282+0000 7f0ed9200640 1 --2- 172.21.15.28:0/3321643471 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ed40c53a0 0x7f0ed4155a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.282+0000 7f0ed9200640 1 -- 172.21.15.28:0/3321643471 >> 172.21.15.28:0/3321643471 conn(0x7f0ed4069e80 msgr2=0x7f0ed406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.283+0000 7f0ed9200640 1 -- 172.21.15.28:0/3321643471 shutdown_connections 2024-09-13T16:44:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:49.283+0000 7f0ed9200640 1 -- 172.21.15.28:0/3321643471 wait complete. 2024-09-13T16:44:52.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:44:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.467+0000 7f4245a00640 1 Processor -- start 2024-09-13T16:44:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.467+0000 7f4245a00640 1 -- start start 2024-09-13T16:44:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.468+0000 7f4245a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 0x7f4240158890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.468+0000 7f4245a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42400c61d0 0x7f4240152720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.468+0000 7f4245a00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4240069e80 con 0x7f424006b4a0 2024-09-13T16:44:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.468+0000 7f4245a00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42400c6730 con 0x7f42400c5df0 2024-09-13T16:44:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.468+0000 7f4245a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f424006a120 con 0x7f42400c61d0 2024-09-13T16:44:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.468+0000 7f4237400640 1 -- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42400c61d0 msgr2=0x7f4240152720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3301/0 2024-09-13T16:44:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.468+0000 7f4237400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42400c61d0 0x7f4240152720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:44:52.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.468+0000 7f423fe00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 0x7f4240158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:52.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.469+0000 7f423fe00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 0x7f4240158890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:50360/0 (socket says 172.21.15.28:50360) 2024-09-13T16:44:52.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.469+0000 7f4245000640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f424006b4a0 0x7f42400c8690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:45554/0 (socket says 172.21.15.28:45554) 2024-09-13T16:44:52.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.469+0000 7f423fe00640 1 -- 172.21.15.28:0/1069697327 learned_addr learned my addr 172.21.15.28:0/1069697327 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:52.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.469+0000 7f423fe00640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 msgr2=0x7f4240158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:44:52.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.469+0000 7f423fe00640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 msgr2=0x7f4240158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:44:52.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.469+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 0x7f4240158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:44:52.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.469+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 0x7f4240158890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:44:52.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.669+0000 7f4237400640 1 -- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42400c61d0 msgr2=0x7f4240152720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3301/0 2024-09-13T16:44:52.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.669+0000 7f4237400640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42400c61d0 0x7f4240152720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T16:44:52.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.670+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 0x7f4240158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:52.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.670+0000 7f423fe00640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 msgr2=0x7f4240158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:44:52.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.670+0000 7f423fe00640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 msgr2=0x7f4240158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:44:52.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.670+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 0x7f4240158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:44:52.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:52.670+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 0x7f4240158890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T16:44:53.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:53.070+0000 7f4237400640 1 -- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42400c61d0 msgr2=0x7f4240152720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3301/0 2024-09-13T16:44:53.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:53.070+0000 7f4237400640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42400c61d0 0x7f4240152720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T16:44:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:53.071+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 0x7f4240158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:53.071+0000 7f423fe00640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 msgr2=0x7f4240158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:44:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:53.071+0000 7f423fe00640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 msgr2=0x7f4240158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:44:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:53.071+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 0x7f4240158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:44:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:53.071+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 0x7f4240158890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T16:44:53.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:53.871+0000 7f4237400640 1 -- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42400c61d0 msgr2=0x7f4240152720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3301/0 2024-09-13T16:44:53.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:53.871+0000 7f4237400640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42400c61d0 0x7f4240152720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T16:44:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:53.873+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 0x7f4240158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:53.873+0000 7f423fe00640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 msgr2=0x7f4240158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:44:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:53.873+0000 7f423fe00640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 msgr2=0x7f4240158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:44:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:53.873+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 0x7f4240158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:44:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:53.873+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 0x7f4240158890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T16:44:55.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.468+0000 7f423e000640 1 -- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42400c61d0 msgr2=0x7f4240152720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:44:55.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.468+0000 7f423e000640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42400c61d0 0x7f4240152720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:55.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.468+0000 7f423e000640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 msgr2=0x7f4240158890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:44:55.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.468+0000 7f423e000640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.191:3300/0 conn(0x7f42400c5df0 0x7f4240158890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:55.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.468+0000 7f423e000640 1 -- 172.21.15.28:0/1069697327 >> v1:172.21.15.191:6789/0 conn(0x7f424006b4a0 legacy=0x7f42400c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:55.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.468+0000 7f423e000640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 0x7f421c005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:55.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.468+0000 7f423e000640 1 -- 172.21.15.28:0/1069697327 --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424006a400 con 0x7f421c006250 2024-09-13T16:44:55.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.468+0000 7f423e000640 1 -- 172.21.15.28:0/1069697327 --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f424006a120 con 0x7f421c0036f0 2024-09-13T16:44:55.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.469+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 0x7f421c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:55.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.469+0000 7f423fe00640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 msgr2=0x7f421c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:44:55.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.469+0000 7f423fe00640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 msgr2=0x7f421c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:44:55.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.469+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 0x7f421c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:44:55.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.469+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 0x7f421c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:44:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.670+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 0x7f421c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.670+0000 7f423fe00640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 msgr2=0x7f421c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:44:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.670+0000 7f423fe00640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 msgr2=0x7f421c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:44:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.670+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 0x7f421c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:44:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:55.670+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 0x7f421c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T16:44:56.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.071+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 0x7f421c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:56.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.071+0000 7f423fe00640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 msgr2=0x7f421c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:44:56.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.071+0000 7f423fe00640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 msgr2=0x7f421c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:44:56.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.071+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 0x7f421c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:44:56.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.071+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 0x7f421c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T16:44:56.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.873+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 0x7f421c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:56.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.873+0000 7f423fe00640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 msgr2=0x7f421c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:44:56.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.873+0000 7f423fe00640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 msgr2=0x7f421c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:44:56.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.873+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 0x7f421c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:44:56.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.873+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 0x7f421c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T16:44:56.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.947+0000 7f423f400640 1 -- 172.21.15.28:0/1069697327 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723229924 0 0) 0x7f424006a400 con 0x7f421c006250 2024-09-13T16:44:56.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.947+0000 7f423f400640 1 -- 172.21.15.28:0/1069697327 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4224003520 con 0x7f421c006250 2024-09-13T16:44:56.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.948+0000 7f423f400640 1 -- 172.21.15.28:0/1069697327 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1258306114 0 0) 0x7f4224003520 con 0x7f421c006250 2024-09-13T16:44:56.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.948+0000 7f423f400640 1 -- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 msgr2=0x7f421c005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:44:56.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.948+0000 7f423f400640 1 --2- 172.21.15.28:0/1069697327 >> v2:172.21.15.28:3300/0 conn(0x7f421c0036f0 0x7f421c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:56.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.948+0000 7f423f400640 1 -- 172.21.15.28:0/1069697327 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421c00afb0 con 0x7f421c006250 2024-09-13T16:44:56.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.948+0000 7f423f400640 1 -- 172.21.15.28:0/1069697327 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1654382301 0 0) 0x7f4228004340 con 0x7f421c006250 2024-09-13T16:44:56.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.948+0000 7f423f400640 1 -- 172.21.15.28:0/1069697327 >> v1:172.21.15.28:6789/0 conn(0x7f421c006250 legacy=0x7f421c008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:56.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.948+0000 7f4245000640 1 -- 172.21.15.28:0/1069697327 reap_dead start 2024-09-13T16:44:56.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.948+0000 7f423f400640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4224003970 0x7f4224003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:56.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.949+0000 7f423f400640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42240047c0 0x7f4224004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:56.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.949+0000 7f423f400640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42240051d0 0x7f422400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:56.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.949+0000 7f423f400640 1 -- 172.21.15.28:0/1069697327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424006a120 con 0x7f42240051d0 2024-09-13T16:44:56.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.949+0000 7f423f400640 1 -- 172.21.15.28:0/1069697327 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422400c100 con 0x7f4224003970 2024-09-13T16:44:56.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.949+0000 7f423f400640 1 -- 172.21.15.28:0/1069697327 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422400c2c0 con 0x7f42240047c0 2024-09-13T16:44:56.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.949+0000 7f4237400640 1 -- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42240047c0 msgr2=0x7f4224004c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3301/0 2024-09-13T16:44:56.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.949+0000 7f4237400640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42240047c0 0x7f4224004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:44:56.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.949+0000 7f4245000640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42240051d0 0x7f422400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:56.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.949+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4224003970 0x7f4224003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:56.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.950+0000 7f4245000640 1 -- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42240047c0 msgr2=0x7f4224004c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:44:56.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.950+0000 7f4245000640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42240047c0 0x7f4224004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:56.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.950+0000 7f4245000640 1 -- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4224003970 msgr2=0x7f4224003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:56.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.950+0000 7f4245000640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4224003970 0x7f4224003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:56.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.950+0000 7f4245000640 1 -- 172.21.15.28:0/1069697327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4224003520 con 0x7f42240051d0 2024-09-13T16:44:56.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.950+0000 7f423fe00640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4224003970 0x7f4224003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:56.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.951+0000 7f4245000640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42240051d0 0x7f422400b980 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f42400c6df0 tx=0x7f42240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fb75115797cd97e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:56.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.952+0000 7f423f400640 1 -- 172.21.15.28:0/1069697327 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238004d60 con 0x7f42240051d0 2024-09-13T16:44:56.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.952+0000 7f423f400640 1 -- 172.21.15.28:0/1069697327 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423802a980 con 0x7f42240051d0 2024-09-13T16:44:56.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.952+0000 7f423f400640 1 -- 172.21.15.28:0/1069697327 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423802ad30 con 0x7f42240051d0 2024-09-13T16:44:56.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.953+0000 7f4245a00640 1 -- 172.21.15.28:0/1069697327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42240051d0 msgr2=0x7f422400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:56.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.953+0000 7f4245a00640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42240051d0 0x7f422400b980 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f42400c6df0 tx=0x7f42240057f0 comp rx=0 tx=0).stop 2024-09-13T16:44:56.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.953+0000 7f4245a00640 1 -- 172.21.15.28:0/1069697327 shutdown_connections 2024-09-13T16:44:56.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.953+0000 7f4245a00640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42240051d0 0x7f422400b980 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:56.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.953+0000 7f4245a00640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42240047c0 0x7f4224004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:56.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.953+0000 7f4245a00640 1 --2- 172.21.15.28:0/1069697327 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4224003970 0x7f4224003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:56.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.953+0000 7f4245a00640 1 -- 172.21.15.28:0/1069697327 >> 172.21.15.28:0/1069697327 conn(0x7f42400c7390 msgr2=0x7f42400c7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:56.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.954+0000 7f4245a00640 1 -- 172.21.15.28:0/1069697327 shutdown_connections 2024-09-13T16:44:56.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.954+0000 7f4245a00640 1 -- 172.21.15.28:0/1069697327 wait complete. 2024-09-13T16:44:56.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.957+0000 7f4245a00640 1 Processor -- start 2024-09-13T16:44:56.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.957+0000 7f4245a00640 1 -- start start 2024-09-13T16:44:56.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.958+0000 7f4245a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4224003970 0x7f42400c4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:56.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.958+0000 7f4245a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42240047c0 0x7f42400bb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:56.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.958+0000 7f4245a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42240051d0 0x7f42400bbf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:56.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.958+0000 7f4245a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42400bb310 con 0x7f4224003970 2024-09-13T16:44:56.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.958+0000 7f4245a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42401ee090 con 0x7f42240051d0 2024-09-13T16:44:56.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.958+0000 7f4245a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42400c3790 con 0x7f42240047c0 2024-09-13T16:44:56.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.958+0000 7f4237400640 1 -- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42240047c0 msgr2=0x7f42400bb9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.191:3301/0 2024-09-13T16:44:56.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.958+0000 7f4237400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42240047c0 0x7f42400bb9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:44:56.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.958+0000 7f423fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4224003970 0x7f42400c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:56.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.958+0000 7f423fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4224003970 0x7f42400c4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38838/0 (socket says 172.21.15.28:38838) 2024-09-13T16:44:56.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.958+0000 7f423fe00640 1 -- 172.21.15.28:0/4247865854 learned_addr learned my addr 172.21.15.28:0/4247865854 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:56.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.958+0000 7f4245000640 1 --2- 172.21.15.28:0/4247865854 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42240051d0 0x7f42400bbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:56.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.959+0000 7f423fe00640 1 -- 172.21.15.28:0/4247865854 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42240047c0 msgr2=0x7f42400bb9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:44:56.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.959+0000 7f423fe00640 1 --2- 172.21.15.28:0/4247865854 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42240047c0 0x7f42400bb9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:56.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.959+0000 7f423fe00640 1 -- 172.21.15.28:0/4247865854 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42240051d0 msgr2=0x7f42400bbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:56.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.959+0000 7f423fe00640 1 --2- 172.21.15.28:0/4247865854 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42240051d0 0x7f42400bbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:56.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.959+0000 7f423fe00640 1 -- 172.21.15.28:0/4247865854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42400c3a70 con 0x7f4224003970 2024-09-13T16:44:56.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.959+0000 7f4245000640 1 --2- 172.21.15.28:0/4247865854 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42240051d0 0x7f42400bbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:44:56.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.959+0000 7f423fe00640 1 --2- 172.21.15.28:0/4247865854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4224003970 0x7f42400c4830 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f42300169c0 tx=0x7f4230016e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:56.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.959+0000 7f423cc00640 1 -- 172.21.15.28:0/4247865854 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423001e050 con 0x7f4224003970 2024-09-13T16:44:56.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.960+0000 7f423cc00640 1 -- 172.21.15.28:0/4247865854 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423001f040 con 0x7f4224003970 2024-09-13T16:44:56.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.960+0000 7f4245a00640 1 -- 172.21.15.28:0/4247865854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42400c3d50 con 0x7f4224003970 2024-09-13T16:44:56.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.960+0000 7f423cc00640 1 -- 172.21.15.28:0/4247865854 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230015d90 con 0x7f4224003970 2024-09-13T16:44:56.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.960+0000 7f4245a00640 1 -- 172.21.15.28:0/4247865854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42400c4310 con 0x7f4224003970 2024-09-13T16:44:56.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.961+0000 7f4245a00640 1 -- 172.21.15.28:0/4247865854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4240071030 con 0x7f4224003970 2024-09-13T16:44:56.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.963+0000 7f423cc00640 1 -- 172.21.15.28:0/4247865854 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4230004020 con 0x7f4224003970 2024-09-13T16:44:56.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.964+0000 7f423cc00640 1 --2- 172.21.15.28:0/4247865854 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f421c06b490 0x7f421c06d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:56.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.964+0000 7f423cc00640 1 -- 172.21.15.28:0/4247865854 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f423008ae30 con 0x7f4224003970 2024-09-13T16:44:56.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.964+0000 7f4237400640 1 --2- 172.21.15.28:0/4247865854 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f421c06b490 0x7f421c06d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:56.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.965+0000 7f4237400640 1 --2- 172.21.15.28:0/4247865854 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f421c06b490 0x7f421c06d950 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4240152cf0 tx=0x7f42280041c0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:56.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:56.969+0000 7f423cc00640 1 -- 172.21.15.28:0/4247865854 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4230055e60 con 0x7f4224003970 2024-09-13T16:44:57.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.215+0000 7f4245a00640 1 -- 172.21.15.28:0/4247865854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f424006a320 con 0x7f4224003970 2024-09-13T16:44:57.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.216+0000 7f423cc00640 1 -- 172.21.15.28:0/4247865854 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f4230027620 con 0x7f4224003970 2024-09-13T16:44:57.218 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":8,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:44:57.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.218+0000 7f4245a00640 1 -- 172.21.15.28:0/4247865854 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f421c06b490 msgr2=0x7f421c06d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:57.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.218+0000 7f4245a00640 1 --2- 172.21.15.28:0/4247865854 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f421c06b490 0x7f421c06d950 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4240152cf0 tx=0x7f42280041c0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.218+0000 7f4245a00640 1 -- 172.21.15.28:0/4247865854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4224003970 msgr2=0x7f42400c4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:57.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.218+0000 7f4245a00640 1 --2- 172.21.15.28:0/4247865854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4224003970 0x7f42400c4830 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f42300169c0 tx=0x7f4230016e80 comp rx=0 tx=0).stop 2024-09-13T16:44:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.218+0000 7f4245a00640 1 -- 172.21.15.28:0/4247865854 shutdown_connections 2024-09-13T16:44:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.218+0000 7f4245a00640 1 --2- 172.21.15.28:0/4247865854 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f421c06b490 0x7f421c06d950 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.218+0000 7f4245a00640 1 --2- 172.21.15.28:0/4247865854 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42240051d0 0x7f42400bbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.218+0000 7f4245a00640 1 --2- 172.21.15.28:0/4247865854 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f42240047c0 0x7f42400bb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.218+0000 7f4245a00640 1 --2- 172.21.15.28:0/4247865854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4224003970 0x7f42400c4830 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.218+0000 7f4245a00640 1 -- 172.21.15.28:0/4247865854 >> 172.21.15.28:0/4247865854 conn(0x7f42400c7390 msgr2=0x7f42401516a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.218+0000 7f4245a00640 1 -- 172.21.15.28:0/4247865854 shutdown_connections 2024-09-13T16:44:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.218+0000 7f4245a00640 1 -- 172.21.15.28:0/4247865854 wait complete. 2024-09-13T16:44:57.236 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-13T16:44:57.236 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-13T16:44:57.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T16:44:57.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.391+0000 7f293b400640 1 Processor -- start 2024-09-13T16:44:57.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.391+0000 7f293b400640 1 -- start start 2024-09-13T16:44:57.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.392+0000 7f293b400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29340c5140 0x7f2934159800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:57.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.392+0000 7f293b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f29340c73e0 0x7f29340c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:57.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.392+0000 7f293b400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29340c6f50 con 0x7f29340c5a60 2024-09-13T16:44:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.392+0000 7f293b400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29340c6940 con 0x7f29340c73e0 2024-09-13T16:44:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.392+0000 7f293b400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2934159d40 con 0x7f29340c5140 2024-09-13T16:44:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.392+0000 7f2938a00640 1 -- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29340c5140 msgr2=0x7f2934159800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3301/0 2024-09-13T16:44:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.392+0000 7f2938a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29340c5140 0x7f2934159800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:44:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.392+0000 7f2933e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f29340c73e0 0x7f29340c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.393+0000 7f2933e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f29340c73e0 0x7f29340c5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38846/0 (socket says 172.21.15.28:38846) 2024-09-13T16:44:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.393+0000 7f2933e00640 1 -- 172.21.15.28:0/4046334714 learned_addr learned my addr 172.21.15.28:0/4046334714 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.393+0000 7f2939400640 1 --1- 172.21.15.28:0/4046334714 >> v1:172.21.15.28:6789/0 conn(0x7f29340c5a60 0x7f293414f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38238/0 (socket says 172.21.15.28:38238) 2024-09-13T16:44:57.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.393+0000 7f2933e00640 1 -- 172.21.15.28:0/4046334714 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29340c5140 msgr2=0x7f2934159800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:44:57.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.393+0000 7f2933e00640 1 --2- 172.21.15.28:0/4046334714 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29340c5140 0x7f2934159800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.393+0000 7f2933e00640 1 -- 172.21.15.28:0/4046334714 >> v1:172.21.15.28:6789/0 conn(0x7f29340c5a60 legacy=0x7f293414f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:57.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.393+0000 7f2933e00640 1 -- 172.21.15.28:0/4046334714 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29340c1b50 con 0x7f29340c73e0 2024-09-13T16:44:57.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.394+0000 7f2933e00640 1 --2- 172.21.15.28:0/4046334714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29340c73e0 0x7f29340c5520 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2920009870 tx=0x7f292002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715b738990907ec4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:57.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.394+0000 7f2933400640 1 -- 172.21.15.28:0/4046334714 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2920032440 con 0x7f29340c73e0 2024-09-13T16:44:57.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.394+0000 7f2933400640 1 -- 172.21.15.28:0/4046334714 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29200325a0 con 0x7f29340c73e0 2024-09-13T16:44:57.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.394+0000 7f2933400640 1 -- 172.21.15.28:0/4046334714 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2920032930 con 0x7f29340c73e0 2024-09-13T16:44:57.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.395+0000 7f293b400640 1 -- 172.21.15.28:0/4046334714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29340c73e0 msgr2=0x7f29340c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:57.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.395+0000 7f293b400640 1 --2- 172.21.15.28:0/4046334714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29340c73e0 0x7f29340c5520 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2920009870 tx=0x7f292002a1e0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.395+0000 7f293b400640 1 -- 172.21.15.28:0/4046334714 shutdown_connections 2024-09-13T16:44:57.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.395+0000 7f293b400640 1 --2- 172.21.15.28:0/4046334714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29340c73e0 0x7f29340c5520 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.395+0000 7f293b400640 1 --2- 172.21.15.28:0/4046334714 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29340c5140 0x7f2934159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.395+0000 7f293b400640 1 -- 172.21.15.28:0/4046334714 >> 172.21.15.28:0/4046334714 conn(0x7f29340ca6b0 msgr2=0x7f29340caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:57.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.395+0000 7f293b400640 1 -- 172.21.15.28:0/4046334714 shutdown_connections 2024-09-13T16:44:57.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.395+0000 7f293b400640 1 -- 172.21.15.28:0/4046334714 wait complete. 2024-09-13T16:44:57.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.396+0000 7f293b400640 1 Processor -- start 2024-09-13T16:44:57.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.397+0000 7f293b400640 1 -- start start 2024-09-13T16:44:57.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.397+0000 7f293b400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29340c5140 0x7f29341e0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:57.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.397+0000 7f293b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29340c5a60 0x7f29340c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:57.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.397+0000 7f293b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29340c73e0 0x7f29341dec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:57.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.397+0000 7f293b400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2934071fd0 con 0x7f29340c5a60 2024-09-13T16:44:57.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.397+0000 7f293b400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29341e1590 con 0x7f29340c73e0 2024-09-13T16:44:57.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.397+0000 7f293b400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29341e11b0 con 0x7f29340c5140 2024-09-13T16:44:57.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.398+0000 7f2938a00640 1 -- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29340c5140 msgr2=0x7f29341e0c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.191:3301/0 2024-09-13T16:44:57.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.398+0000 7f2938a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29340c5140 0x7f29341e0c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:44:57.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.398+0000 7f2933e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29340c5a60 0x7f29340c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:57.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.398+0000 7f2933e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29340c5a60 0x7f29340c97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38848/0 (socket says 172.21.15.28:38848) 2024-09-13T16:44:57.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.398+0000 7f2933e00640 1 -- 172.21.15.28:0/2558152265 learned_addr learned my addr 172.21.15.28:0/2558152265 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:57.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.398+0000 7f2939400640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29340c73e0 0x7f29341dec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:57.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.398+0000 7f2933e00640 1 -- 172.21.15.28:0/2558152265 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29340c5140 msgr2=0x7f29341e0c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:44:57.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.398+0000 7f2933e00640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f29340c5140 0x7f29341e0c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.398+0000 7f2933e00640 1 -- 172.21.15.28:0/2558152265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29340c73e0 msgr2=0x7f29341dec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:57.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.398+0000 7f2933e00640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f29340c73e0 0x7f29341dec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.398+0000 7f2933e00640 1 -- 172.21.15.28:0/2558152265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29200094d0 con 0x7f29340c5a60 2024-09-13T16:44:57.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.399+0000 7f2933e00640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29340c5a60 0x7f29340c97a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f29200090b0 tx=0x7f29200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:57.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.399+0000 7f2930c00640 1 -- 172.21.15.28:0/2558152265 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2920032440 con 0x7f29340c5a60 2024-09-13T16:44:57.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.399+0000 7f2930c00640 1 -- 172.21.15.28:0/2558152265 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2920030ba0 con 0x7f29340c5a60 2024-09-13T16:44:57.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.399+0000 7f2930c00640 1 -- 172.21.15.28:0/2558152265 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292003b400 con 0x7f29340c5a60 2024-09-13T16:44:57.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.399+0000 7f293b400640 1 -- 172.21.15.28:0/2558152265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2934072200 con 0x7f29340c5a60 2024-09-13T16:44:57.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.399+0000 7f293b400640 1 -- 172.21.15.28:0/2558152265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2934071110 con 0x7f29340c5a60 2024-09-13T16:44:57.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.400+0000 7f293b400640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29341e1850 0x7f29341f1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:57.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.400+0000 7f293b400640 1 -- 172.21.15.28:0/2558152265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2934066c20 con 0x7f29341e1850 2024-09-13T16:44:57.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.400+0000 7f2938a00640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29341e1850 0x7f29341f1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:57.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.401+0000 7f2938a00640 1 -- 172.21.15.28:0/2558152265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f29340c6be0 con 0x7f29341e1850 2024-09-13T16:44:57.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.401+0000 7f2930c00640 1 -- 172.21.15.28:0/2558152265 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2920039040 con 0x7f29340c5a60 2024-09-13T16:44:57.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.401+0000 7f2938a00640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29341e1850 0x7f29341f1630 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2928003930 tx=0x7f292800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:57.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.402+0000 7f2930c00640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f291406abb0 0x7f291406d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:57.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.402+0000 7f2930c00640 1 -- 172.21.15.28:0/2558152265 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29200a9bd0 con 0x7f29340c5a60 2024-09-13T16:44:57.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.402+0000 7f2930c00640 1 -- 172.21.15.28:0/2558152265 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2928003e10 con 0x7f29341e1850 2024-09-13T16:44:57.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.402+0000 7f2930c00640 1 -- 172.21.15.28:0/2558152265 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f2928005070 con 0x7f29341e1850 2024-09-13T16:44:57.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.402+0000 7f2930c00640 1 -- 172.21.15.28:0/2558152265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29341e1850 msgr2=0x7f29341f1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:57.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.402+0000 7f2939400640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f291406abb0 0x7f291406d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:57.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.402+0000 7f2930c00640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29341e1850 0x7f29341f1630 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2928003930 tx=0x7f292800b8c0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.402+0000 7f2930c00640 1 -- 172.21.15.28:0/2558152265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29341e1850 msgr2=0x7f29341f1630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:44:57.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.402+0000 7f2930c00640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29341e1850 0x7f29341f1630 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.403+0000 7f2939400640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f291406abb0 0x7f291406d070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f292c0012e0 tx=0x7f292c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:57.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.409+0000 7f293b400640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29341f1b70 0x7f2934140c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:57.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.409+0000 7f293b400640 1 -- 172.21.15.28:0/2558152265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293406fe80 con 0x7f29341f1b70 2024-09-13T16:44:57.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.409+0000 7f2938a00640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29341f1b70 0x7f2934140c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:57.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.409+0000 7f2938a00640 1 -- 172.21.15.28:0/2558152265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2934149830 con 0x7f29341f1b70 2024-09-13T16:44:57.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.410+0000 7f2938a00640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29341f1b70 0x7f2934140c70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2928003930 tx=0x7f2928005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:57.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.410+0000 7f2930c00640 1 -- 172.21.15.28:0/2558152265 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292801f070 con 0x7f29341f1b70 2024-09-13T16:44:57.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.410+0000 7f2930c00640 1 -- 172.21.15.28:0/2558152265 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+5088 (secure 0 0 0) 0x7f2928010400 con 0x7f29341f1b70 2024-09-13T16:44:57.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.410+0000 7f2930c00640 1 -- 172.21.15.28:0/2558152265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29341f1b70 msgr2=0x7f2934140c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:57.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.410+0000 7f2930c00640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29341f1b70 0x7f2934140c70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2928003930 tx=0x7f2928005a20 comp rx=0 tx=0).stop 2024-09-13T16:44:57.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.410+0000 7f2930c00640 1 -- 172.21.15.28:0/2558152265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29341f1b70 msgr2=0x7f2934140c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:44:57.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.410+0000 7f2930c00640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29341f1b70 0x7f2934140c70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.413 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:44:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:44:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:44:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "leader", 2024-09-13T16:44:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 8, 2024-09-13T16:44:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:44:57.413 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:44:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: 1 2024-09-13T16:44:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 0, 2024-09-13T16:44:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:44:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:44:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:44:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:44:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:44:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:44:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:44:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:44:57.414 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:44:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:44:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:44:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:44:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:44:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:44:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:44:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:44:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:44:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:44:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:44:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:44:57.415 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:44:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:44:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:44:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:44:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:44:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:44:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:44:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:44:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:44:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:44:57.416 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:44:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:44:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:44:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:44:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:44:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:44:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:44:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:44:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:44:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:44:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:44:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:44:57.417 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:44:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:44:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:44:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:44:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:44:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:44:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:44:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:44:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:44:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:44:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:44:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.418 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:44:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:44:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:44:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:44:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:44:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:44:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:44:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:44:57.419 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:44:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:44:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:44:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:44:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:44:57.420 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:44:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:44:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:44:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:44:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:44:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:44:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:44:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.421 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:44:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:44:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:44:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:44:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:44:57.422 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:44:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:44:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:44:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:44:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:44:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:44:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:44:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:44:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:44:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:44:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:44:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:44:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:44:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:44:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:44:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:44:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:44:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:44:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:44:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:44:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:44:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4 2024-09-13T16:44:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:44:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x2f018fb87aa4aafe", 2024-09-13T16:44:57.426 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "luminous", 2024-09-13T16:44:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:44:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4 2024-09-13T16:44:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": [ 2024-09-13T16:44:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:57.427 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:44:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.428 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:44:57.428 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:44:57.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.411+0000 7f293b400640 1 -- 172.21.15.28:0/2558152265 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f291406abb0 msgr2=0x7f291406d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:57.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.411+0000 7f293b400640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f291406abb0 0x7f291406d070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f292c0012e0 tx=0x7f292c009040 comp rx=0 tx=0).stop 2024-09-13T16:44:57.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.411+0000 7f293b400640 1 -- 172.21.15.28:0/2558152265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29340c5a60 msgr2=0x7f29340c97a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:57.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.411+0000 7f293b400640 1 --2- 172.21.15.28:0/2558152265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29340c5a60 0x7f29340c97a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f29200090b0 tx=0x7f29200046c0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.411+0000 7f2939400640 1 -- 172.21.15.28:0/2558152265 reap_dead start 2024-09-13T16:44:57.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.411+0000 7f2939400640 1 -- 172.21.15.28:0/2558152265 reap_dead start 2024-09-13T16:44:57.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.411+0000 7f293b400640 1 -- 172.21.15.28:0/2558152265 shutdown_connections 2024-09-13T16:44:57.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.411+0000 7f293b400640 1 -- 172.21.15.28:0/2558152265 >> 172.21.15.28:0/2558152265 conn(0x7f29340ca6b0 msgr2=0x7f29340caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:57.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.411+0000 7f293b400640 1 -- 172.21.15.28:0/2558152265 shutdown_connections 2024-09-13T16:44:57.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.411+0000 7f293b400640 1 -- 172.21.15.28:0/2558152265 wait complete. 2024-09-13T16:44:57.430 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-13T16:44:57.430 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-13T16:44:57.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T16:44:57.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.585+0000 7f3380000640 1 Processor -- start 2024-09-13T16:44:57.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.585+0000 7f3380000640 1 -- start start 2024-09-13T16:44:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.586+0000 7f3380000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f33780c5140 0x7f3378159800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.586+0000 7f3380000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33780c73e0 0x7f33780c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.586+0000 7f3380000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f33780c5a60 0x7f337814f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.587+0000 7f3380000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33780c6f50 con 0x7f33780c5140 2024-09-13T16:44:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.587+0000 7f3380000640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33780c6940 con 0x7f33780c5a60 2024-09-13T16:44:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.587+0000 7f3380000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3378159d40 con 0x7f33780c73e0 2024-09-13T16:44:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.587+0000 7f337cc00640 1 -- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33780c73e0 msgr2=0x7f33780c5520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3301/0 2024-09-13T16:44:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.587+0000 7f337cc00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33780c73e0 0x7f33780c5520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:44:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.587+0000 7f337d600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f33780c5140 0x7f3378159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.587+0000 7f337d600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f33780c5140 0x7f3378159800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38874/0 (socket says 172.21.15.28:38874) 2024-09-13T16:44:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.587+0000 7f337e000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f33780c5a60 0x7f337814f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.587+0000 7f337d600640 1 -- 172.21.15.28:0/2511823205 learned_addr learned my addr 172.21.15.28:0/2511823205 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:57.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.587+0000 7f337d600640 1 -- 172.21.15.28:0/2511823205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33780c73e0 msgr2=0x7f33780c5520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:44:57.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.587+0000 7f337d600640 1 --2- 172.21.15.28:0/2511823205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33780c73e0 0x7f33780c5520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.587+0000 7f337d600640 1 -- 172.21.15.28:0/2511823205 >> v2:172.21.15.191:3300/0 conn(0x7f33780c5a60 msgr2=0x7f337814f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:44:57.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.587+0000 7f337d600640 1 --2- 172.21.15.28:0/2511823205 >> v2:172.21.15.191:3300/0 conn(0x7f33780c5a60 0x7f337814f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.587+0000 7f337d600640 1 -- 172.21.15.28:0/2511823205 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33780c1b50 con 0x7f33780c5140 2024-09-13T16:44:57.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.587+0000 7f337e000640 1 --2- 172.21.15.28:0/2511823205 >> v2:172.21.15.191:3300/0 conn(0x7f33780c5a60 0x7f337814f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:44:57.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.588+0000 7f337d600640 1 --2- 172.21.15.28:0/2511823205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33780c5140 0x7f3378159800 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f33780c8380 tx=0x7f336c02a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1c7a7332e4de60 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:57.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.589+0000 7f3377e00640 1 -- 172.21.15.28:0/2511823205 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c032440 con 0x7f33780c5140 2024-09-13T16:44:57.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.589+0000 7f3377e00640 1 -- 172.21.15.28:0/2511823205 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336c0325a0 con 0x7f33780c5140 2024-09-13T16:44:57.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.589+0000 7f3377e00640 1 -- 172.21.15.28:0/2511823205 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c032930 con 0x7f33780c5140 2024-09-13T16:44:57.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.589+0000 7f3380000640 1 -- 172.21.15.28:0/2511823205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33780c5140 msgr2=0x7f3378159800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.589+0000 7f3380000640 1 --2- 172.21.15.28:0/2511823205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33780c5140 0x7f3378159800 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f33780c8380 tx=0x7f336c02a160 comp rx=0 tx=0).stop 2024-09-13T16:44:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.589+0000 7f3380000640 1 -- 172.21.15.28:0/2511823205 shutdown_connections 2024-09-13T16:44:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.589+0000 7f3380000640 1 --2- 172.21.15.28:0/2511823205 >> v2:172.21.15.191:3300/0 conn(0x7f33780c5a60 0x7f337814f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.589+0000 7f3380000640 1 --2- 172.21.15.28:0/2511823205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33780c73e0 0x7f33780c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.589+0000 7f3380000640 1 --2- 172.21.15.28:0/2511823205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33780c5140 0x7f3378159800 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.589+0000 7f3380000640 1 -- 172.21.15.28:0/2511823205 >> 172.21.15.28:0/2511823205 conn(0x7f33780ca6b0 msgr2=0x7f33780caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.589+0000 7f3380000640 1 -- 172.21.15.28:0/2511823205 shutdown_connections 2024-09-13T16:44:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.589+0000 7f3380000640 1 -- 172.21.15.28:0/2511823205 wait complete. 2024-09-13T16:44:57.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.591+0000 7f3380000640 1 Processor -- start 2024-09-13T16:44:57.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.591+0000 7f3380000640 1 -- start start 2024-09-13T16:44:57.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.592+0000 7f3380000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33780c5140 0x7f33781ed090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:57.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.592+0000 7f3380000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33780c5a60 0x7f33780c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:57.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.592+0000 7f3380000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33780c73e0 0x7f3378158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:57.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.592+0000 7f3380000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33781ed5d0 con 0x7f33780c5140 2024-09-13T16:44:57.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.592+0000 7f3380000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3378141a10 con 0x7f33780c5a60 2024-09-13T16:44:57.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.592+0000 7f3380000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3378141cf0 con 0x7f33780c73e0 2024-09-13T16:44:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.592+0000 7f337d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33780c5140 0x7f33781ed090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.592+0000 7f337e000640 1 -- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33780c73e0 msgr2=0x7f3378158080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.191:3301/0 2024-09-13T16:44:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.592+0000 7f337e000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33780c73e0 0x7f3378158080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:44:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.592+0000 7f337d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33780c5140 0x7f33781ed090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38890/0 (socket says 172.21.15.28:38890) 2024-09-13T16:44:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.592+0000 7f337d600640 1 -- 172.21.15.28:0/3960324991 learned_addr learned my addr 172.21.15.28:0/3960324991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:44:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.593+0000 7f337cc00640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33780c5a60 0x7f33780c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.593+0000 7f337d600640 1 -- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33780c73e0 msgr2=0x7f3378158080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:44:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.593+0000 7f337d600640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33780c73e0 0x7f3378158080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.593+0000 7f337d600640 1 -- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33780c5a60 msgr2=0x7f33780c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.593+0000 7f337d600640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33780c5a60 0x7f33780c97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.593+0000 7f337d600640 1 -- 172.21.15.28:0/3960324991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3370009a70 con 0x7f33780c5140 2024-09-13T16:44:57.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.593+0000 7f337d600640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33780c5140 0x7f33781ed090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f336c000c00 tx=0x7f336c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:57.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.593+0000 7f3375600640 1 -- 172.21.15.28:0/3960324991 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c032440 con 0x7f33780c5140 2024-09-13T16:44:57.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.593+0000 7f3375600640 1 -- 172.21.15.28:0/3960324991 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336c030980 con 0x7f33780c5140 2024-09-13T16:44:57.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.593+0000 7f3375600640 1 -- 172.21.15.28:0/3960324991 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c030cf0 con 0x7f33780c5140 2024-09-13T16:44:57.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.593+0000 7f3380000640 1 -- 172.21.15.28:0/3960324991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336c0094d0 con 0x7f33780c5140 2024-09-13T16:44:57.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.593+0000 7f3380000640 1 -- 172.21.15.28:0/3960324991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33781422b0 con 0x7f33780c5140 2024-09-13T16:44:57.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.594+0000 7f3380000640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33781434f0 0x7f33781424a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:57.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.594+0000 7f3380000640 1 -- 172.21.15.28:0/3960324991 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3378066c20 con 0x7f33781434f0 2024-09-13T16:44:57.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.595+0000 7f337cc00640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33781434f0 0x7f33781424a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:57.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.595+0000 7f3375600640 1 -- 172.21.15.28:0/3960324991 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f336c039040 con 0x7f33780c5140 2024-09-13T16:44:57.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.595+0000 7f337cc00640 1 -- 172.21.15.28:0/3960324991 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33780c6be0 con 0x7f33781434f0 2024-09-13T16:44:57.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.596+0000 7f337cc00640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33781434f0 0x7f33781424a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f33780c12f0 tx=0x7f336400c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:57.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.596+0000 7f3375600640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f335806abb0 0x7f335806d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:57.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.596+0000 7f3375600640 1 -- 172.21.15.28:0/3960324991 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f336c0a9b70 con 0x7f33780c5140 2024-09-13T16:44:57.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.596+0000 7f337e000640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f335806abb0 0x7f335806d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:57.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.596+0000 7f3375600640 1 -- 172.21.15.28:0/3960324991 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3364003db0 con 0x7f33781434f0 2024-09-13T16:44:57.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.596+0000 7f3375600640 1 -- 172.21.15.28:0/3960324991 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f3364005070 con 0x7f33781434f0 2024-09-13T16:44:57.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.596+0000 7f3375600640 1 -- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33781434f0 msgr2=0x7f33781424a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:57.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.596+0000 7f3375600640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33781434f0 0x7f33781424a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f33780c12f0 tx=0x7f336400c950 comp rx=0 tx=0).stop 2024-09-13T16:44:57.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.596+0000 7f3375600640 1 -- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33781434f0 msgr2=0x7f33781424a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:44:57.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.596+0000 7f3375600640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33781434f0 0x7f33781424a0 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f33780c12f0 tx=0x7f336400c950 comp rx=0 tx=0).stop 2024-09-13T16:44:57.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.597+0000 7f337e000640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f335806abb0 0x7f335806d070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3378157ce0 tx=0x7f337000b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:57.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.603+0000 7f3380000640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33781585c0 0x7f337813fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:44:57.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.603+0000 7f3380000640 1 -- 172.21.15.28:0/3960324991 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337814e8a0 con 0x7f33781585c0 2024-09-13T16:44:57.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.604+0000 7f337cc00640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33781585c0 0x7f337813fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:44:57.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.604+0000 7f337cc00640 1 -- 172.21.15.28:0/3960324991 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3378066f40 con 0x7f33781585c0 2024-09-13T16:44:57.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.605+0000 7f337cc00640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33781585c0 0x7f337813fcf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3364006fd0 tx=0x7f33640059c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:44:57.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.605+0000 7f3375600640 1 -- 172.21.15.28:0/3960324991 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3364021070 con 0x7f33781585c0 2024-09-13T16:44:57.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.605+0000 7f3375600640 1 -- 172.21.15.28:0/3960324991 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4943 (secure 0 0 0) 0x7f3364005800 con 0x7f33781585c0 2024-09-13T16:44:57.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.605+0000 7f3375600640 1 -- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33781585c0 msgr2=0x7f337813fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:57.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.605+0000 7f3375600640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33781585c0 0x7f337813fcf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3364006fd0 tx=0x7f33640059c0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.605+0000 7f3375600640 1 -- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33781585c0 msgr2=0x7f337813fcf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:44:57.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.605+0000 7f3375600640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33781585c0 0x7f337813fcf0 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3364006fd0 tx=0x7f33640059c0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.608 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:44:57.608 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:44:57.608 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:44:57.608 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:44:57.608 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 8, 2024-09-13T16:44:57.608 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:44:57.608 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:44:57.608 INFO:teuthology.orchestra.run.smithi028.stdout: 1 2024-09-13T16:44:57.608 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:57.608 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 0, 2024-09-13T16:44:57.608 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:44:57.609 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:44:57.609 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:44:57.609 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:44:57.609 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:44:57.609 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:44:57.609 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:44:57.609 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:44:57.609 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:44:57.609 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:44:57.609 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:44:57.609 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:44:57.609 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:44:57.610 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:57.610 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:44:57.610 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:44:57.610 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:44:57.610 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:44:57.610 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:44:57.610 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:44:57.610 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:44:57.610 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:44:57.610 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:44:57.610 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:44:57.610 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:44:57.610 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:44:57.611 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:57.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.611 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:44:57.611 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:44:57.611 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:44:57.611 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:44:57.611 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:44:57.611 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:44:57.611 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:44:57.611 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:44:57.611 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:44:57.611 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:44:57.612 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:44:57.612 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:44:57.612 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:44:57.612 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:44:57.612 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:44:57.612 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:44:57.612 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:44:57.612 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:44:57.613 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:44:57.613 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:44:57.613 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:44:57.613 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:44:57.613 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:44:57.613 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:44:57.613 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:44:57.613 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:44:57.613 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:44:57.613 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:57.613 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:44:57.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.613 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:44:57.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.614 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:44:57.614 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:44:57.614 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:44:57.614 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:44:57.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.614 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:44:57.614 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:44:57.614 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:57.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.615 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:44:57.615 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:44:57.615 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:57.615 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.615 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:57.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.615 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:44:57.615 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:44:57.615 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:44:57.615 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:44:57.615 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:44:57.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.616 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:44:57.616 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:44:57.616 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:44:57.616 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:44:57.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.616 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:44:57.616 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:44:57.617 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:57.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:44:57.617 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:44:57.617 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:57.617 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.617 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:57.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.618 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:44:57.618 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:44:57.618 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:44:57.618 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:44:57.618 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:44:57.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.618 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:44:57.618 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:44:57.618 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:44:57.618 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:44:57.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.618 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:44:57.618 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:44:57.619 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:57.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.619 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:44:57.619 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:44:57.619 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:44:57.619 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.619 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:57.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.619 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:44:57.619 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:44:57.619 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:44:57.619 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:44:57.620 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:44:57.620 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.620 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:57.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.620 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:44:57.620 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:44:57.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.620 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:57.620 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:57.620 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:44:57.620 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.620 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:57.621 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:44:57.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.621 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:57.621 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:57.621 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:44:57.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.621 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:57.621 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:44:57.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.621 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:57.621 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:57.621 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4 2024-09-13T16:44:57.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.622 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:57.622 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:44:57.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.622 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:57.622 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:57.622 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:44:57.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.622 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:44:57.622 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": [ 2024-09-13T16:44:57.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:44:57.622 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:44:57.622 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:44:57.623 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:44:57.623 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:44:57.623 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:44:57.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:44:57.623 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:44:57.623 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:44:57.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.606+0000 7f3380000640 1 -- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f335806abb0 msgr2=0x7f335806d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:57.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.606+0000 7f3380000640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f335806abb0 0x7f335806d070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3378157ce0 tx=0x7f337000b040 comp rx=0 tx=0).stop 2024-09-13T16:44:57.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.606+0000 7f3380000640 1 -- 172.21.15.28:0/3960324991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33780c5140 msgr2=0x7f33781ed090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:44:57.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.606+0000 7f3380000640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33780c5140 0x7f33781ed090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f336c000c00 tx=0x7f336c0046c0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.606+0000 7f337e000640 1 -- 172.21.15.28:0/3960324991 reap_dead start 2024-09-13T16:44:57.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.606+0000 7f3380000640 1 -- 172.21.15.28:0/3960324991 shutdown_connections 2024-09-13T16:44:57.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.606+0000 7f3380000640 1 --2- 172.21.15.28:0/3960324991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33780c5140 0x7f33781ed090 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:44:57.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.606+0000 7f3380000640 1 -- 172.21.15.28:0/3960324991 >> 172.21.15.28:0/3960324991 conn(0x7f33780ca6b0 msgr2=0x7f3378149910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:44:57.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.606+0000 7f3380000640 1 -- 172.21.15.28:0/3960324991 shutdown_connections 2024-09-13T16:44:57.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:44:57.606+0000 7f3380000640 1 -- 172.21.15.28:0/3960324991 wait complete. 2024-09-13T16:44:57.625 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-13T16:44:57.625 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-13T16:45:03.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:45:03.483 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:45:17.626 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-13T16:45:17.626 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-13T16:45:17.627 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-13T16:45:17.627 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-13T16:45:17.630 INFO:tasks.ceph.mon.c:Started 2024-09-13T16:45:17.630 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T16:45:17.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:45:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.792+0000 7fde6cc00640 1 Processor -- start 2024-09-13T16:45:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.793+0000 7fde6cc00640 1 -- start start 2024-09-13T16:45:17.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.794+0000 7fde6cc00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde680c5060 0x7fde680c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:17.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.794+0000 7fde6cc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde680c5df0 con 0x7fde68156610 2024-09-13T16:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.794+0000 7fde6cc00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde680c2ab0 con 0x7fde680c5440 2024-09-13T16:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.794+0000 7fde6cc00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde680c2d70 con 0x7fde680c5060 2024-09-13T16:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.794+0000 7fde67400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde680c5060 0x7fde680c44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.794+0000 7fde66a00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fde680c5440 0x7fde680c2300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:59156/0 (socket says 172.21.15.28:59156) 2024-09-13T16:45:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.794+0000 7fde66a00640 1 -- 172.21.15.28:0/1886222531 learned_addr learned my addr 172.21.15.28:0/1886222531 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:45:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde67400640 1 -- 172.21.15.28:0/1886222531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde680c5060 msgr2=0x7fde680c44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:45:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde67400640 1 -- 172.21.15.28:0/1886222531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde680c5060 msgr2=0x7fde680c44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:45:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde67400640 1 --2- 172.21.15.28:0/1886222531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde680c5060 0x7fde680c44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde67400640 1 --2- 172.21.15.28:0/1886222531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde680c5060 0x7fde680c44d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde66000640 1 -- 172.21.15.28:0/1886222531 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169100229 0 0) 0x7fde680c2ab0 con 0x7fde680c5440 2024-09-13T16:45:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde66000640 1 -- 172.21.15.28:0/1886222531 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde4c003520 con 0x7fde680c5440 2024-09-13T16:45:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde66000640 1 -- 172.21.15.28:0/1886222531 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738811053 0 0) 0x7fde680c5df0 con 0x7fde68156610 2024-09-13T16:45:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde66000640 1 -- 172.21.15.28:0/1886222531 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde680c2ab0 con 0x7fde68156610 2024-09-13T16:45:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde66000640 1 -- 172.21.15.28:0/1886222531 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1503024509 0 0) 0x7fde4c003520 con 0x7fde680c5440 2024-09-13T16:45:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde66000640 1 -- 172.21.15.28:0/1886222531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde680c5060 msgr2=0x7fde680c44d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:45:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde66000640 1 --2- 172.21.15.28:0/1886222531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde680c5060 0x7fde680c44d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde66000640 1 -- 172.21.15.28:0/1886222531 >> v1:172.21.15.28:6789/0 conn(0x7fde68156610 legacy=0x7fde68150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde66000640 1 -- 172.21.15.28:0/1886222531 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde680c6040 con 0x7fde680c5440 2024-09-13T16:45:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde66000640 1 -- 172.21.15.28:0/1886222531 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1654382301 0 0) 0x7fde54001670 con 0x7fde680c5440 2024-09-13T16:45:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde66000640 1 -- 172.21.15.28:0/1886222531 >> v1:172.21.15.191:6789/0 conn(0x7fde680c5440 legacy=0x7fde680c2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde66000640 1 --2- 172.21.15.28:0/1886222531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde4c003b00 0x7fde4c003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde66000640 1 --2- 172.21.15.28:0/1886222531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde4c0047c0 0x7fde4c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:17.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.795+0000 7fde66000640 1 --2- 172.21.15.28:0/1886222531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde680c5060 0x7fde4c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:17.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.796+0000 7fde66000640 1 -- 172.21.15.28:0/1886222531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde680c2d70 con 0x7fde4c003b00 2024-09-13T16:45:17.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.796+0000 7fde66000640 1 -- 172.21.15.28:0/1886222531 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c00ba80 con 0x7fde4c0047c0 2024-09-13T16:45:17.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.796+0000 7fde66000640 1 -- 172.21.15.28:0/1886222531 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c00bc40 con 0x7fde680c5060 2024-09-13T16:45:17.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.796+0000 7fde66a00640 1 --2- 172.21.15.28:0/1886222531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde4c0047c0 0x7fde4c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:17.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.796+0000 7fde67400640 1 --2- 172.21.15.28:0/1886222531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde4c003b00 0x7fde4c003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:17.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.796+0000 7fde67e00640 1 --2- 172.21.15.28:0/1886222531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde680c5060 0x7fde4c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:17.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.796+0000 7fde66a00640 1 -- 172.21.15.28:0/1886222531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde680c5060 msgr2=0x7fde4c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:45:17.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.796+0000 7fde66a00640 1 --2- 172.21.15.28:0/1886222531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde680c5060 0x7fde4c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:17.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.796+0000 7fde66a00640 1 -- 172.21.15.28:0/1886222531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde4c003b00 msgr2=0x7fde4c003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:45:17.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.796+0000 7fde66a00640 1 --2- 172.21.15.28:0/1886222531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde4c003b00 0x7fde4c003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:17.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.796+0000 7fde66a00640 1 -- 172.21.15.28:0/1886222531 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde680c6040 con 0x7fde4c0047c0 2024-09-13T16:45:17.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.797+0000 7fde67400640 1 --2- 172.21.15.28:0/1886222531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde4c003b00 0x7fde4c003f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:45:17.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.798+0000 7fde66a00640 1 --2- 172.21.15.28:0/1886222531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde4c0047c0 0x7fde4c004c40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fde5400c210 tx=0x7fde4c0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47fe5043f297501a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:17.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.798+0000 7fde66000640 1 -- 172.21.15.28:0/1886222531 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54005ad0 con 0x7fde4c0047c0 2024-09-13T16:45:17.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.798+0000 7fde66000640 1 -- 172.21.15.28:0/1886222531 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5402e070 con 0x7fde4c0047c0 2024-09-13T16:45:17.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.798+0000 7fde66000640 1 -- 172.21.15.28:0/1886222531 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54005e50 con 0x7fde4c0047c0 2024-09-13T16:45:17.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.799+0000 7fde6cc00640 1 -- 172.21.15.28:0/1886222531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde4c0047c0 msgr2=0x7fde4c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:17.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.799+0000 7fde6cc00640 1 --2- 172.21.15.28:0/1886222531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde4c0047c0 0x7fde4c004c40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fde5400c210 tx=0x7fde4c0057f0 comp rx=0 tx=0).stop 2024-09-13T16:45:17.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.799+0000 7fde67e00640 1 -- 172.21.15.28:0/1886222531 reap_dead start 2024-09-13T16:45:17.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.799+0000 7fde6cc00640 1 -- 172.21.15.28:0/1886222531 shutdown_connections 2024-09-13T16:45:17.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.799+0000 7fde6cc00640 1 -- 172.21.15.28:0/1886222531 >> 172.21.15.28:0/1886222531 conn(0x7fde68069e80 msgr2=0x7fde6806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:17.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.799+0000 7fde6cc00640 1 -- 172.21.15.28:0/1886222531 shutdown_connections 2024-09-13T16:45:17.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.799+0000 7fde6cc00640 1 -- 172.21.15.28:0/1886222531 wait complete. 2024-09-13T16:45:17.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.801+0000 7fde6cc00640 1 Processor -- start 2024-09-13T16:45:17.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.801+0000 7fde6cc00640 1 -- start start 2024-09-13T16:45:17.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.803+0000 7fde6cc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde4c0047c0 0x7fde68144360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:17.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.803+0000 7fde6cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde681400f0 0x7fde68140550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:17.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.803+0000 7fde6cc00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6813ee10 0x7fde6813f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:17.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.803+0000 7fde6cc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde68140a90 con 0x7fde681400f0 2024-09-13T16:45:17.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.803+0000 7fde6cc00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6813f7d0 con 0x7fde4c0047c0 2024-09-13T16:45:17.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.803+0000 7fde6cc00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde68140e40 con 0x7fde6813ee10 2024-09-13T16:45:17.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.803+0000 7fde66a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde681400f0 0x7fde68140550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:17.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.803+0000 7fde67400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde4c0047c0 0x7fde68144360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:17.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.803+0000 7fde67400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde4c0047c0 0x7fde68144360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:44150/0 (socket says 172.21.15.28:44150) 2024-09-13T16:45:17.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.803+0000 7fde67400640 1 -- 172.21.15.28:0/346309558 learned_addr learned my addr 172.21.15.28:0/346309558 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:17.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.803+0000 7fde67e00640 1 --2- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6813ee10 0x7fde6813f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:17.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.803+0000 7fde67e00640 1 -- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6813ee10 msgr2=0x7fde6813f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-13T16:45:17.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.803+0000 7fde67e00640 1 -- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6813ee10 msgr2=0x7fde6813f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T16:45:17.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.803+0000 7fde67e00640 1 --2- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6813ee10 0x7fde6813f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:17.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.803+0000 7fde67e00640 1 --2- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6813ee10 0x7fde6813f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:17.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.803+0000 7fde67400640 1 -- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6813ee10 msgr2=0x7fde6813f290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:45:17.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.803+0000 7fde67400640 1 --2- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6813ee10 0x7fde6813f290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:17.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.804+0000 7fde67400640 1 -- 172.21.15.28:0/346309558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde681400f0 msgr2=0x7fde68140550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:17.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.804+0000 7fde67400640 1 --2- 172.21.15.28:0/346309558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde681400f0 0x7fde68140550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:17.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.804+0000 7fde67400640 1 -- 172.21.15.28:0/346309558 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde68141b30 con 0x7fde4c0047c0 2024-09-13T16:45:17.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.804+0000 7fde66a00640 1 --2- 172.21.15.28:0/346309558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde681400f0 0x7fde68140550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:45:17.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.804+0000 7fde67400640 1 --2- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde4c0047c0 0x7fde68144360 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fde580109b0 tx=0x7fde58010e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:17.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.804+0000 7fde5f400640 1 -- 172.21.15.28:0/346309558 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5800eb40 con 0x7fde4c0047c0 2024-09-13T16:45:17.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.804+0000 7fde5f400640 1 -- 172.21.15.28:0/346309558 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5800eca0 con 0x7fde4c0047c0 2024-09-13T16:45:17.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.804+0000 7fde6cc00640 1 -- 172.21.15.28:0/346309558 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde68154f80 con 0x7fde4c0047c0 2024-09-13T16:45:17.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.804+0000 7fde5f400640 1 -- 172.21.15.28:0/346309558 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58016480 con 0x7fde4c0047c0 2024-09-13T16:45:17.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.804+0000 7fde6cc00640 1 -- 172.21.15.28:0/346309558 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde68142060 con 0x7fde4c0047c0 2024-09-13T16:45:17.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.806+0000 7fde6cc00640 1 -- 172.21.15.28:0/346309558 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde40005250 con 0x7fde4c0047c0 2024-09-13T16:45:17.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.806+0000 7fde5f400640 1 -- 172.21.15.28:0/346309558 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fde58016600 con 0x7fde4c0047c0 2024-09-13T16:45:17.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.808+0000 7fde5f400640 1 --2- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fde4806ac00 0x7fde4806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:17.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.808+0000 7fde5f400640 1 -- 172.21.15.28:0/346309558 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fde58089ca0 con 0x7fde4c0047c0 2024-09-13T16:45:17.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.808+0000 7fde66a00640 1 --2- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fde4806ac00 0x7fde4806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:17.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.809+0000 7fde66a00640 1 --2- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fde4806ac00 0x7fde4806d0c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fde54033000 tx=0x7fde5402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:17.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:17.812+0000 7fde5f400640 1 -- 172.21.15.28:0/346309558 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fde58054d60 con 0x7fde4c0047c0 2024-09-13T16:45:18.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.218+0000 7fde6cc00640 1 -- 172.21.15.28:0/346309558 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fde40005a80 con 0x7fde4c0047c0 2024-09-13T16:45:18.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.219+0000 7fde5f400640 1 -- 172.21.15.28:0/346309558 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7fde58058380 con 0x7fde4c0047c0 2024-09-13T16:45:18.220 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:45:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.222+0000 7fde6cc00640 1 -- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fde4806ac00 msgr2=0x7fde4806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.222+0000 7fde6cc00640 1 --2- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fde4806ac00 0x7fde4806d0c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fde54033000 tx=0x7fde5402e000 comp rx=0 tx=0).stop 2024-09-13T16:45:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.222+0000 7fde6cc00640 1 -- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde4c0047c0 msgr2=0x7fde68144360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.222+0000 7fde6cc00640 1 --2- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde4c0047c0 0x7fde68144360 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fde580109b0 tx=0x7fde58010e70 comp rx=0 tx=0).stop 2024-09-13T16:45:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.222+0000 7fde6cc00640 1 -- 172.21.15.28:0/346309558 shutdown_connections 2024-09-13T16:45:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.222+0000 7fde6cc00640 1 --2- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fde4806ac00 0x7fde4806d0c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.222+0000 7fde6cc00640 1 --2- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde6813ee10 0x7fde6813f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.222+0000 7fde6cc00640 1 --2- 172.21.15.28:0/346309558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde681400f0 0x7fde68140550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.222+0000 7fde6cc00640 1 --2- 172.21.15.28:0/346309558 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde4c0047c0 0x7fde68144360 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.222+0000 7fde6cc00640 1 -- 172.21.15.28:0/346309558 >> 172.21.15.28:0/346309558 conn(0x7fde68069e80 msgr2=0x7fde68155a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.222+0000 7fde6cc00640 1 -- 172.21.15.28:0/346309558 shutdown_connections 2024-09-13T16:45:18.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.222+0000 7fde6cc00640 1 -- 172.21.15.28:0/346309558 wait complete. 2024-09-13T16:45:18.250 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T16:45:18.250 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-13T16:45:18.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T16:45:18.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.440+0000 7f8bfbe00640 1 Processor -- start 2024-09-13T16:45:18.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.441+0000 7f8bfbe00640 1 -- start start 2024-09-13T16:45:18.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.443+0000 7f8bfbe00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8bf415a440 0x7f8bf40c5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.443+0000 7f8bfbe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf40c1d80 con 0x7f8bf415a060 2024-09-13T16:45:18.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.443+0000 7f8bfbe00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf415a920 con 0x7f8bf406dcb0 2024-09-13T16:45:18.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.443+0000 7f8bfbe00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf415ac00 con 0x7f8bf415a440 2024-09-13T16:45:18.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.443+0000 7f8bf9400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8bf415a060 0x7f8bf406d5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:32968/0 (socket says 172.21.15.28:32968) 2024-09-13T16:45:18.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.443+0000 7f8bf9400640 1 -- 172.21.15.28:0/372670607 learned_addr learned my addr 172.21.15.28:0/372670607 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:45:18.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.444+0000 7f8bf3e00640 1 -- 172.21.15.28:0/372670607 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466182409 0 0) 0x7f8bf40c1d80 con 0x7f8bf415a060 2024-09-13T16:45:18.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.444+0000 7f8bf8a00640 1 --2- 172.21.15.28:0/372670607 >> v2:172.21.15.191:3300/0 conn(0x7f8bf415a440 0x7f8bf40c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.444+0000 7f8bf3e00640 1 -- 172.21.15.28:0/372670607 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bdc003520 con 0x7f8bf415a060 2024-09-13T16:45:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.445+0000 7f8bf3e00640 1 -- 172.21.15.28:0/372670607 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1739691426 0 0) 0x7f8bdc003520 con 0x7f8bf415a060 2024-09-13T16:45:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.445+0000 7f8bf3e00640 1 -- 172.21.15.28:0/372670607 >> v2:172.21.15.191:3300/0 conn(0x7f8bf415a440 msgr2=0x7f8bf40c5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:45:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.445+0000 7f8bf3e00640 1 --2- 172.21.15.28:0/372670607 >> v2:172.21.15.191:3300/0 conn(0x7f8bf415a440 0x7f8bf40c5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.445+0000 7f8bf3e00640 1 -- 172.21.15.28:0/372670607 >> v1:172.21.15.191:6789/0 conn(0x7f8bf406dcb0 legacy=0x7f8bf41504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.445+0000 7f8bf3e00640 1 -- 172.21.15.28:0/372670607 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf40c2d50 con 0x7f8bf415a060 2024-09-13T16:45:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.445+0000 7f8bf3e00640 1 -- 172.21.15.28:0/372670607 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1654382301 0 0) 0x7f8be8001670 con 0x7f8bf415a060 2024-09-13T16:45:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.445+0000 7f8bf3e00640 1 -- 172.21.15.28:0/372670607 >> v1:172.21.15.28:6789/0 conn(0x7f8bf415a060 legacy=0x7f8bf406d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.445+0000 7f8bf3e00640 1 --2- 172.21.15.28:0/372670607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bdc003970 0x7f8bdc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.445+0000 7f8bf3e00640 1 --2- 172.21.15.28:0/372670607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bdc0047c0 0x7f8bdc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.445+0000 7f8bf3e00640 1 --2- 172.21.15.28:0/372670607 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bdc005180 0x7f8bdc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.445+0000 7f8bf3e00640 1 -- 172.21.15.28:0/372670607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf415ac00 con 0x7f8bdc003970 2024-09-13T16:45:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.445+0000 7f8bf3e00640 1 -- 172.21.15.28:0/372670607 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc00c0d0 con 0x7f8bdc0047c0 2024-09-13T16:45:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.445+0000 7f8bf3e00640 1 -- 172.21.15.28:0/372670607 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc00c290 con 0x7f8bdc005180 2024-09-13T16:45:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.445+0000 7f8bf8a00640 1 --2- 172.21.15.28:0/372670607 >> v2:172.21.15.191:3300/0 conn(0x7f8bf415a440 0x7f8bf40c5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:45:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.445+0000 7f8bf8a00640 1 --2- 172.21.15.28:0/372670607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bdc0047c0 0x7f8bdc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.445+0000 7f8bf9400640 1 --2- 172.21.15.28:0/372670607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bdc003970 0x7f8bdc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.446+0000 7f8bf8a00640 1 -- 172.21.15.28:0/372670607 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bdc005180 msgr2=0x7f8bdc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:45:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.446+0000 7f8bf8a00640 1 --2- 172.21.15.28:0/372670607 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bdc005180 0x7f8bdc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.446+0000 7f8bf8a00640 1 -- 172.21.15.28:0/372670607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bdc003970 msgr2=0x7f8bdc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:45:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.446+0000 7f8bf8a00640 1 --2- 172.21.15.28:0/372670607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bdc003970 0x7f8bdc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.446+0000 7f8bf8a00640 1 -- 172.21.15.28:0/372670607 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf40c2d50 con 0x7f8bdc0047c0 2024-09-13T16:45:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.446+0000 7f8bf9400640 1 --2- 172.21.15.28:0/372670607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bdc003970 0x7f8bdc003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:45:18.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.446+0000 7f8bf9e00640 1 -- 172.21.15.28:0/372670607 reap_dead start 2024-09-13T16:45:18.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.447+0000 7f8bf8a00640 1 --2- 172.21.15.28:0/372670607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bdc0047c0 0x7f8bdc004c40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8bf415a850 tx=0x7f8bdc0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ae170e9e1f427ee server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:18.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.447+0000 7f8bf3e00640 1 -- 172.21.15.28:0/372670607 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be002ac80 con 0x7f8bdc0047c0 2024-09-13T16:45:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.448+0000 7f8bf3e00640 1 -- 172.21.15.28:0/372670607 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be002ade0 con 0x7f8bdc0047c0 2024-09-13T16:45:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.448+0000 7f8bf3e00640 1 -- 172.21.15.28:0/372670607 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be0032640 con 0x7f8bdc0047c0 2024-09-13T16:45:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.448+0000 7f8bfbe00640 1 -- 172.21.15.28:0/372670607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bdc0047c0 msgr2=0x7f8bdc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.448+0000 7f8bfbe00640 1 --2- 172.21.15.28:0/372670607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bdc0047c0 0x7f8bdc004c40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8bf415a850 tx=0x7f8bdc0057f0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.448+0000 7f8bfbe00640 1 -- 172.21.15.28:0/372670607 shutdown_connections 2024-09-13T16:45:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.448+0000 7f8bfbe00640 1 --2- 172.21.15.28:0/372670607 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bdc0047c0 0x7f8bdc004c40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.448+0000 7f8bfbe00640 1 -- 172.21.15.28:0/372670607 >> 172.21.15.28:0/372670607 conn(0x7f8bf40ca6b0 msgr2=0x7f8bf40caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:18.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.449+0000 7f8bfbe00640 1 -- 172.21.15.28:0/372670607 shutdown_connections 2024-09-13T16:45:18.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.449+0000 7f8bfbe00640 1 -- 172.21.15.28:0/372670607 wait complete. 2024-09-13T16:45:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.451+0000 7f8bfbe00640 1 Processor -- start 2024-09-13T16:45:18.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.452+0000 7f8bfbe00640 1 -- start start 2024-09-13T16:45:18.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.453+0000 7f8bfbe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bdc0047c0 0x7f8bf41579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.453+0000 7f8bfbe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bf4157ee0 0x7f8bf4071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.453+0000 7f8bfbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf4159950 0x7f8bf4071580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.453+0000 7f8bfbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf41eb490 con 0x7f8bf4159950 2024-09-13T16:45:18.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.453+0000 7f8bfbe00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf4159220 con 0x7f8bf4157ee0 2024-09-13T16:45:18.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.453+0000 7f8bfbe00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf41517b0 con 0x7f8bdc0047c0 2024-09-13T16:45:18.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.453+0000 7f8bf9400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bdc0047c0 0x7f8bf41579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.454+0000 7f8bf9400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bdc0047c0 0x7f8bf41579a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:56650/0 (socket says 172.21.15.28:56650) 2024-09-13T16:45:18.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.454+0000 7f8bf9400640 1 -- 172.21.15.28:0/2784914774 learned_addr learned my addr 172.21.15.28:0/2784914774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:18.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.454+0000 7f8bf8a00640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bf4157ee0 0x7f8bf4071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.454+0000 7f8bf9400640 1 -- 172.21.15.28:0/2784914774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bdc0047c0 msgr2=0x7f8bf41579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-13T16:45:18.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.454+0000 7f8bf9400640 1 -- 172.21.15.28:0/2784914774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bdc0047c0 msgr2=0x7f8bf41579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T16:45:18.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.454+0000 7f8bf9400640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bdc0047c0 0x7f8bf41579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:18.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.454+0000 7f8bf9e00640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf4159950 0x7f8bf4071580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.454+0000 7f8bf9400640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bdc0047c0 0x7f8bf41579a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:18.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.454+0000 7f8bf8a00640 1 -- 172.21.15.28:0/2784914774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bdc0047c0 msgr2=0x7f8bf41579a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:45:18.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.454+0000 7f8bf8a00640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bdc0047c0 0x7f8bf41579a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.454+0000 7f8bf8a00640 1 -- 172.21.15.28:0/2784914774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf4159950 msgr2=0x7f8bf4071580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.454+0000 7f8bf8a00640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf4159950 0x7f8bf4071580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.454+0000 7f8bf8a00640 1 -- 172.21.15.28:0/2784914774 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf4154c80 con 0x7f8bf4157ee0 2024-09-13T16:45:18.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.455+0000 7f8bf8a00640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bf4157ee0 0x7f8bf4071040 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc00c260 tx=0x7f8be0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:18.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.455+0000 7f8bf1600640 1 -- 172.21.15.28:0/2784914774 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be003e650 con 0x7f8bf4157ee0 2024-09-13T16:45:18.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.455+0000 7f8bf1600640 1 -- 172.21.15.28:0/2784914774 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be003e7b0 con 0x7f8bf4157ee0 2024-09-13T16:45:18.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.455+0000 7f8bfbe00640 1 -- 172.21.15.28:0/2784914774 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf41eaef0 con 0x7f8bf4157ee0 2024-09-13T16:45:18.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.455+0000 7f8bf1600640 1 -- 172.21.15.28:0/2784914774 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be003eb20 con 0x7f8bf4157ee0 2024-09-13T16:45:18.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.455+0000 7f8bfbe00640 1 -- 172.21.15.28:0/2784914774 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf41eb1b0 con 0x7f8bf4157ee0 2024-09-13T16:45:18.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.457+0000 7f8bfbe00640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf41510f0 0x7f8bf4148c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.457+0000 7f8bfbe00640 1 -- 172.21.15.28:0/2784914774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf4066c20 con 0x7f8bf41510f0 2024-09-13T16:45:18.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.457+0000 7f8bf9400640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf41510f0 0x7f8bf4148c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.459+0000 7f8bf1600640 1 -- 172.21.15.28:0/2784914774 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8be003ec80 con 0x7f8bf4157ee0 2024-09-13T16:45:18.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.459+0000 7f8bf9400640 1 -- 172.21.15.28:0/2784914774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8bf40c5220 con 0x7f8bf41510f0 2024-09-13T16:45:18.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.460+0000 7f8bf9400640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf41510f0 0x7f8bf4148c60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8be8000fb0 tx=0x7f8be8014670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:18.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.460+0000 7f8bf1600640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8bd406ac30 0x7f8bd406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.461+0000 7f8bf9e00640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8bd406ac30 0x7f8bd406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.461+0000 7f8bf1600640 1 -- 172.21.15.28:0/2784914774 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8be00a9e30 con 0x7f8bf4157ee0 2024-09-13T16:45:18.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.461+0000 7f8bf1600640 1 -- 172.21.15.28:0/2784914774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be8018600 con 0x7f8bf41510f0 2024-09-13T16:45:18.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.461+0000 7f8bf1600640 1 -- 172.21.15.28:0/2784914774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f8be8018760 con 0x7f8bf41510f0 2024-09-13T16:45:18.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.462+0000 7f8bf1600640 1 -- 172.21.15.28:0/2784914774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf41510f0 msgr2=0x7f8bf4148c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.462+0000 7f8bf1600640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf41510f0 0x7f8bf4148c60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8be8000fb0 tx=0x7f8be8014670 comp rx=0 tx=0).stop 2024-09-13T16:45:18.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.462+0000 7f8bf1600640 1 -- 172.21.15.28:0/2784914774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf41510f0 msgr2=0x7f8bf4148c60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:45:18.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.462+0000 7f8bf1600640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf41510f0 0x7f8bf4148c60 secure :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8be8000fb0 tx=0x7f8be8014670 comp rx=0 tx=0).stop 2024-09-13T16:45:18.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.462+0000 7f8bf9e00640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8bd406ac30 0x7f8bd406d0f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8bf406aed0 tx=0x7f8bec009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:18.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.474+0000 7f8bfbe00640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf41491a0 0x7f8bf41ed9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.474+0000 7f8bfbe00640 1 -- 172.21.15.28:0/2784914774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf40c3310 con 0x7f8bf41491a0 2024-09-13T16:45:18.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.475+0000 7f8bf9400640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf41491a0 0x7f8bf41ed9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.476+0000 7f8bf9400640 1 -- 172.21.15.28:0/2784914774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8bf40c19d0 con 0x7f8bf41491a0 2024-09-13T16:45:18.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.476+0000 7f8bf9400640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf41491a0 0x7f8bf41ed9e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8be8000fb0 tx=0x7f8be8000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:18.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.477+0000 7f8bf1600640 1 -- 172.21.15.28:0/2784914774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be8021620 con 0x7f8bf41491a0 2024-09-13T16:45:18.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.477+0000 7f8bf1600640 1 -- 172.21.15.28:0/2784914774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+5100 (secure 0 0 0) 0x7f8be8021780 con 0x7f8bf41491a0 2024-09-13T16:45:18.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.477+0000 7f8bf1600640 1 -- 172.21.15.28:0/2784914774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf41491a0 msgr2=0x7f8bf41ed9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.477+0000 7f8bf1600640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf41491a0 0x7f8bf41ed9e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8be8000fb0 tx=0x7f8be8000fe0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.479 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:45:18.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:45:18.480 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:45:18.480 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "leader", 2024-09-13T16:45:18.480 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 10, 2024-09-13T16:45:18.480 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:45:18.480 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:45:18.480 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:45:18.480 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:45:18.480 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:18.480 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 0, 2024-09-13T16:45:18.480 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:45:18.480 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:45:18.480 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:45:18.480 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:18.481 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:18.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:18.481 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:18.481 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:18.481 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:18.481 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:18.481 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:18.481 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:18.481 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:18.481 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:18.481 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:45:18.481 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:45:18.482 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:18.482 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:18.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:18.482 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:18.482 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:18.482 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:18.482 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:18.482 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:18.482 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:18.482 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:18.482 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:18.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.482 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:45:18.483 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:45:18.483 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:45:18.483 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:45:18.483 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:45:18.483 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:45:18.483 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:45:18.483 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:45:18.483 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:45:18.483 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:45:18.483 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:45:18.483 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:45:18.483 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:45:18.484 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:45:18.484 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:45:18.484 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:45:18.484 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:45:18.484 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:18.484 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:18.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:18.484 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:18.484 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:18.484 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:18.484 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:18.484 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:18.484 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:18.484 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:18.485 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:18.485 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:45:18.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:45:18.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.485 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:45:18.485 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:45:18.485 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:18.485 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:18.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.485 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:18.485 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:45:18.485 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:18.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.486 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:18.486 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:45:18.486 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:18.486 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.486 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:18.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.486 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:45:18.486 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:45:18.486 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:18.486 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:18.486 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:18.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.487 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:45:18.487 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:45:18.487 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:18.487 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:18.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.487 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:18.487 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:45:18.487 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:18.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.487 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:18.487 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:45:18.487 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:18.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.488 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:18.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.488 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:45:18.488 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:45:18.488 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:18.488 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:18.488 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:18.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.488 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:45:18.488 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:45:18.488 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:18.488 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:18.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:18.489 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:45:18.489 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:18.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:18.489 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:45:18.489 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:18.489 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.489 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:18.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.489 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:45:18.489 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:45:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:45:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:45:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:45:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:45:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:45:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:45:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4 2024-09-13T16:45:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:45:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x2f018fb87aa4aafe", 2024-09-13T16:45:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "luminous", 2024-09-13T16:45:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:45:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:45:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": [ 2024-09-13T16:45:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:45:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:45:18.493 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:45:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.478+0000 7f8bf1600640 1 -- 172.21.15.28:0/2784914774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf41491a0 msgr2=0x7f8bf41ed9e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:45:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.478+0000 7f8bf1600640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf41491a0 0x7f8bf41ed9e0 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8be8000fb0 tx=0x7f8be8000fe0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.478+0000 7f8bfbe00640 1 -- 172.21.15.28:0/2784914774 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8bd406ac30 msgr2=0x7f8bd406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.478+0000 7f8bfbe00640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8bd406ac30 0x7f8bd406d0f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8bf406aed0 tx=0x7f8bec009040 comp rx=0 tx=0).stop 2024-09-13T16:45:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.478+0000 7f8bfbe00640 1 -- 172.21.15.28:0/2784914774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bf4157ee0 msgr2=0x7f8bf4071040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.478+0000 7f8bfbe00640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bf4157ee0 0x7f8bf4071040 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc00c260 tx=0x7f8be0004730 comp rx=0 tx=0).stop 2024-09-13T16:45:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.478+0000 7f8bf9e00640 1 -- 172.21.15.28:0/2784914774 reap_dead start 2024-09-13T16:45:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.478+0000 7f8bfbe00640 1 -- 172.21.15.28:0/2784914774 shutdown_connections 2024-09-13T16:45:18.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.478+0000 7f8bfbe00640 1 --2- 172.21.15.28:0/2784914774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bf4157ee0 0x7f8bf4071040 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.478+0000 7f8bfbe00640 1 -- 172.21.15.28:0/2784914774 >> 172.21.15.28:0/2784914774 conn(0x7f8bf40ca6b0 msgr2=0x7f8bf40c1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:18.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.479+0000 7f8bfbe00640 1 -- 172.21.15.28:0/2784914774 shutdown_connections 2024-09-13T16:45:18.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.479+0000 7f8bfbe00640 1 -- 172.21.15.28:0/2784914774 wait complete. 2024-09-13T16:45:18.496 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:45:18.496 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-13T16:45:18.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T16:45:18.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.663+0000 7f4147200640 1 Processor -- start 2024-09-13T16:45:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.663+0000 7f4147200640 1 -- start start 2024-09-13T16:45:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.665+0000 7f4147200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f414806bbe0 0x7f41480c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.665+0000 7f4147200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f414806bfc0 0x7f4148160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.665+0000 7f4147200640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4148070370 con 0x7f4148156b80 2024-09-13T16:45:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.665+0000 7f4147200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4148160750 con 0x7f414806bbe0 2024-09-13T16:45:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.665+0000 7f4147200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41481609e0 con 0x7f414806bfc0 2024-09-13T16:45:18.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.665+0000 7f4145e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f414806bbe0 0x7f41480c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.665+0000 7f4145e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f414806bbe0 0x7f41480c67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45468/0 (socket says 172.21.15.28:45468) 2024-09-13T16:45:18.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.665+0000 7f4145e00640 1 -- 172.21.15.28:0/395211761 learned_addr learned my addr 172.21.15.28:0/395211761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:18.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.665+0000 7f4145400640 1 --2- 172.21.15.28:0/395211761 >> v2:172.21.15.191:3300/0 conn(0x7f414806bfc0 0x7f4148160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.666+0000 7f4144a00640 1 -- 172.21.15.28:0/395211761 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698775186 0 0) 0x7f4148070370 con 0x7f4148156b80 2024-09-13T16:45:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.666+0000 7f4144a00640 1 -- 172.21.15.28:0/395211761 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f412c003520 con 0x7f4148156b80 2024-09-13T16:45:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.666+0000 7f4145e00640 1 -- 172.21.15.28:0/395211761 >> v2:172.21.15.191:3300/0 conn(0x7f414806bfc0 msgr2=0x7f4148160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:45:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.666+0000 7f4145e00640 1 --2- 172.21.15.28:0/395211761 >> v2:172.21.15.191:3300/0 conn(0x7f414806bfc0 0x7f4148160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.666+0000 7f4145e00640 1 -- 172.21.15.28:0/395211761 >> v1:172.21.15.191:6789/0 conn(0x7f4148156b80 legacy=0x7f4148070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.666+0000 7f4145e00640 1 -- 172.21.15.28:0/395211761 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4148160cc0 con 0x7f414806bbe0 2024-09-13T16:45:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.666+0000 7f4145400640 1 --2- 172.21.15.28:0/395211761 >> v2:172.21.15.191:3300/0 conn(0x7f414806bfc0 0x7f4148160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:45:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.667+0000 7f4145e00640 1 --2- 172.21.15.28:0/395211761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f414806bbe0 0x7f41480c67e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4148071eb0 tx=0x7f413802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b6eb5ac152b2f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.668+0000 7f4144a00640 1 -- 172.21.15.28:0/395211761 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4138032440 con 0x7f414806bbe0 2024-09-13T16:45:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.668+0000 7f4144a00640 1 -- 172.21.15.28:0/395211761 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41380325a0 con 0x7f414806bbe0 2024-09-13T16:45:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.668+0000 7f4144a00640 1 -- 172.21.15.28:0/395211761 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4138032930 con 0x7f414806bbe0 2024-09-13T16:45:18.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.670+0000 7f4147200640 1 -- 172.21.15.28:0/395211761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f414806bbe0 msgr2=0x7f41480c67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.670+0000 7f4147200640 1 --2- 172.21.15.28:0/395211761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f414806bbe0 0x7f41480c67e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4148071eb0 tx=0x7f413802a270 comp rx=0 tx=0).stop 2024-09-13T16:45:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.670+0000 7f4147200640 1 -- 172.21.15.28:0/395211761 shutdown_connections 2024-09-13T16:45:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.670+0000 7f4147200640 1 --2- 172.21.15.28:0/395211761 >> v2:172.21.15.191:3300/0 conn(0x7f414806bfc0 0x7f4148160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.670+0000 7f4147200640 1 --2- 172.21.15.28:0/395211761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f414806bbe0 0x7f41480c67e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.670+0000 7f4147200640 1 -- 172.21.15.28:0/395211761 >> 172.21.15.28:0/395211761 conn(0x7f41480ca2f0 msgr2=0x7f41480ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.670+0000 7f4147200640 1 -- 172.21.15.28:0/395211761 shutdown_connections 2024-09-13T16:45:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.671+0000 7f4147200640 1 -- 172.21.15.28:0/395211761 wait complete. 2024-09-13T16:45:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.672+0000 7f4147200640 1 Processor -- start 2024-09-13T16:45:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.673+0000 7f4147200640 1 -- start start 2024-09-13T16:45:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.674+0000 7f4147200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f414806bbe0 0x7f41480c2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.674+0000 7f4147200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f414806bfc0 0x7f41480c2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.674+0000 7f4147200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4148156b80 0x7f41481491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.674+0000 7f4147200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41480c2a20 con 0x7f4148156b80 2024-09-13T16:45:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.674+0000 7f4147200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414815f800 con 0x7f414806bbe0 2024-09-13T16:45:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.674+0000 7f4147200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41481f3540 con 0x7f414806bfc0 2024-09-13T16:45:18.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.674+0000 7f4146800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4148156b80 0x7f41481491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.674+0000 7f4146800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4148156b80 0x7f41481491a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45480/0 (socket says 172.21.15.28:45480) 2024-09-13T16:45:18.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.674+0000 7f4146800640 1 -- 172.21.15.28:0/3161180834 learned_addr learned my addr 172.21.15.28:0/3161180834 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:18.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.675+0000 7f4145400640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f414806bfc0 0x7f41480c2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.675+0000 7f4145400640 1 -- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f414806bfc0 msgr2=0x7f41480c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-13T16:45:18.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.675+0000 7f4145400640 1 -- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f414806bfc0 msgr2=0x7f41480c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T16:45:18.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.675+0000 7f4145e00640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f414806bbe0 0x7f41480c2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.675+0000 7f4145400640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f414806bfc0 0x7f41480c2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:18.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.675+0000 7f4145400640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f414806bfc0 0x7f41480c2e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.675+0000 7f4146800640 1 -- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f414806bfc0 msgr2=0x7f41480c2e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:45:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.675+0000 7f4146800640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f414806bfc0 0x7f41480c2e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.675+0000 7f4146800640 1 -- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f414806bbe0 msgr2=0x7f41480c2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.675+0000 7f4146800640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f414806bbe0 0x7f41480c2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.675+0000 7f4146800640 1 -- 172.21.15.28:0/3161180834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4134009400 con 0x7f4148156b80 2024-09-13T16:45:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.676+0000 7f4146800640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4148156b80 0x7f41481491a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f414000c710 tx=0x7f414000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.677+0000 7f413e000640 1 -- 172.21.15.28:0/3161180834 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4140003b70 con 0x7f4148156b80 2024-09-13T16:45:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.677+0000 7f413e000640 1 -- 172.21.15.28:0/3161180834 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4140003cd0 con 0x7f4148156b80 2024-09-13T16:45:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.677+0000 7f413e000640 1 -- 172.21.15.28:0/3161180834 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414000aa50 con 0x7f4148156b80 2024-09-13T16:45:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.678+0000 7f4147200640 1 -- 172.21.15.28:0/3161180834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41380094d0 con 0x7f4148156b80 2024-09-13T16:45:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.678+0000 7f4147200640 1 -- 172.21.15.28:0/3161180834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4148154610 con 0x7f4148156b80 2024-09-13T16:45:18.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.679+0000 7f4147200640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41480c3380 0x7f41481496e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.679+0000 7f4147200640 1 -- 172.21.15.28:0/3161180834 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4148066c20 con 0x7f41480c3380 2024-09-13T16:45:18.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.679+0000 7f4145e00640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41480c3380 0x7f41481496e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.680+0000 7f4145e00640 1 -- 172.21.15.28:0/3161180834 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f41480708a0 con 0x7f41480c3380 2024-09-13T16:45:18.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.681+0000 7f4145e00640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41480c3380 0x7f41481496e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f41480cb2c0 tx=0x7f41380046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:18.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.681+0000 7f413e000640 1 -- 172.21.15.28:0/3161180834 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f414000abb0 con 0x7f4148156b80 2024-09-13T16:45:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.683+0000 7f413e000640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f412806ac30 0x7f412806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.683+0000 7f413e000640 1 -- 172.21.15.28:0/3161180834 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4140089120 con 0x7f4148156b80 2024-09-13T16:45:18.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.683+0000 7f4145400640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f412806ac30 0x7f412806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.683+0000 7f413e000640 1 -- 172.21.15.28:0/3161180834 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4138032440 con 0x7f41480c3380 2024-09-13T16:45:18.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.683+0000 7f413e000640 1 -- 172.21.15.28:0/3161180834 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f413803c880 con 0x7f41480c3380 2024-09-13T16:45:18.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.683+0000 7f413e000640 1 -- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41480c3380 msgr2=0x7f41481496e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.684+0000 7f413e000640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41480c3380 0x7f41481496e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f41480cb2c0 tx=0x7f41380046c0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.684+0000 7f413e000640 1 -- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41480c3380 msgr2=0x7f41481496e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:45:18.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.684+0000 7f413e000640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41480c3380 0x7f41481496e0 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f41480cb2c0 tx=0x7f41380046c0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.685+0000 7f4145400640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f412806ac30 0x7f412806d0f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4148150c60 tx=0x7f4134008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:18.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.697+0000 7f4147200640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41480c3e00 0x7f414815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.697+0000 7f4147200640 1 -- 172.21.15.28:0/3161180834 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41480705c0 con 0x7f41480c3e00 2024-09-13T16:45:18.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.697+0000 7f4145e00640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41480c3e00 0x7f414815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.697+0000 7f4145e00640 1 -- 172.21.15.28:0/3161180834 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f41480c4e10 con 0x7f41480c3e00 2024-09-13T16:45:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.698+0000 7f4145e00640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41480c3e00 0x7f414815e5a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f413802a2b0 tx=0x7f4138009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.698+0000 7f413e000640 1 -- 172.21.15.28:0/3161180834 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4138052480 con 0x7f41480c3e00 2024-09-13T16:45:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.699+0000 7f413e000640 1 -- 172.21.15.28:0/3161180834 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4955 (secure 0 0 0) 0x7f413803b3d0 con 0x7f41480c3e00 2024-09-13T16:45:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.699+0000 7f413e000640 1 -- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41480c3e00 msgr2=0x7f414815e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.699+0000 7f413e000640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41480c3e00 0x7f414815e5a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f413802a2b0 tx=0x7f4138009240 comp rx=0 tx=0).stop 2024-09-13T16:45:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.699+0000 7f413e000640 1 -- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41480c3e00 msgr2=0x7f414815e5a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:45:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.699+0000 7f413e000640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41480c3e00 0x7f414815e5a0 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f413802a2b0 tx=0x7f4138009240 comp rx=0 tx=0).stop 2024-09-13T16:45:18.701 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:45:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:45:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:45:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:45:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 10, 2024-09-13T16:45:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:45:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:45:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:45:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:45:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:18.701 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 0, 2024-09-13T16:45:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:45:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:45:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:45:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:18.702 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:45:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:45:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:18.703 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:45:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:45:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:45:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:45:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:45:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:45:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:45:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:45:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:45:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:45:18.704 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:45:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:45:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:45:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:45:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:45:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:45:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:45:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:45:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:45:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:45:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:45:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.706 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:45:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:45:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:45:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:45:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:18.707 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:45:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:45:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:45:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:45:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:45:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:45:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.709 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:45:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:45:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:45:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:45:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.710 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:45:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:45:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:45:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:45:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:18.711 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:45:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:45:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:45:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:45:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.712 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4 2024-09-13T16:45:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:45:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:45:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": [ 2024-09-13T16:45:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:18.713 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:45:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:45:18.714 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:45:18.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.699+0000 7f4147200640 1 -- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f412806ac30 msgr2=0x7f412806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.699+0000 7f4147200640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f412806ac30 0x7f412806d0f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4148150c60 tx=0x7f4134008040 comp rx=0 tx=0).stop 2024-09-13T16:45:18.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.699+0000 7f4147200640 1 -- 172.21.15.28:0/3161180834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4148156b80 msgr2=0x7f41481491a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.699+0000 7f4147200640 1 --2- 172.21.15.28:0/3161180834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4148156b80 0x7f41481491a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f414000c710 tx=0x7f414000cbd0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.699+0000 7f4146800640 1 -- 172.21.15.28:0/3161180834 reap_dead start 2024-09-13T16:45:18.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.699+0000 7f4146800640 1 -- 172.21.15.28:0/3161180834 reap_dead start 2024-09-13T16:45:18.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.700+0000 7f4147200640 1 -- 172.21.15.28:0/3161180834 shutdown_connections 2024-09-13T16:45:18.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.700+0000 7f4147200640 1 -- 172.21.15.28:0/3161180834 >> 172.21.15.28:0/3161180834 conn(0x7f41480ca2f0 msgr2=0x7f41481513a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:18.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.700+0000 7f4147200640 1 -- 172.21.15.28:0/3161180834 shutdown_connections 2024-09-13T16:45:18.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.700+0000 7f4147200640 1 -- 172.21.15.28:0/3161180834 wait complete. 2024-09-13T16:45:18.722 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:45:18.723 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-13T16:45:18.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T16:45:18.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.905+0000 7f8f38600640 1 Processor -- start 2024-09-13T16:45:18.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.907+0000 7f8f38600640 1 -- start start 2024-09-13T16:45:18.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.909+0000 7f8f38600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 0x7f8f300c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.909+0000 7f8f38600640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8f300ccd30 0x7f8f30155a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.909+0000 7f8f38600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f300c8080 con 0x7f8f300cc950 2024-09-13T16:45:18.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.909+0000 7f8f38600640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f3006bac0 con 0x7f8f300ccd30 2024-09-13T16:45:18.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.909+0000 7f8f38600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3015c3e0 con 0x7f8f300c7010 2024-09-13T16:45:18.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.909+0000 7f8f35c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8f300cc950 0x7f8f30158ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52646/0 (socket says 172.21.15.28:52646) 2024-09-13T16:45:18.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.909+0000 7f8f35c00640 1 -- 172.21.15.28:0/2681181649 learned_addr learned my addr 172.21.15.28:0/2681181649 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:45:18.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.909+0000 7f8f36600640 1 --2- 172.21.15.28:0/2681181649 >> v2:172.21.15.191:3300/0 conn(0x7f8f300ccd30 0x7f8f30155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.909+0000 7f8f35200640 1 --2- 172.21.15.28:0/2681181649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 0x7f8f300c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.910+0000 7f8f35200640 1 -- 172.21.15.28:0/2681181649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 msgr2=0x7f8f300c79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T16:45:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.910+0000 7f8f35200640 1 -- 172.21.15.28:0/2681181649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 msgr2=0x7f8f300c79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:45:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.910+0000 7f8f35200640 1 --2- 172.21.15.28:0/2681181649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 0x7f8f300c79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.910+0000 7f8f35200640 1 --2- 172.21.15.28:0/2681181649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 0x7f8f300c79d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.911+0000 7f8f36600640 1 -- 172.21.15.28:0/2681181649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 msgr2=0x7f8f300c79d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:45:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.911+0000 7f8f36600640 1 --2- 172.21.15.28:0/2681181649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 0x7f8f300c79d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.911+0000 7f8f36600640 1 -- 172.21.15.28:0/2681181649 >> v1:172.21.15.28:6789/0 conn(0x7f8f300cc950 legacy=0x7f8f30158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.911+0000 7f8f36600640 1 -- 172.21.15.28:0/2681181649 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3015c6c0 con 0x7f8f300ccd30 2024-09-13T16:45:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.912+0000 7f8f36600640 1 --2- 172.21.15.28:0/2681181649 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f300ccd30 0x7f8f30155a90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8f28009d90 tx=0x7f8f2802a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8185607030e980c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.912+0000 7f8f2fe00640 1 -- 172.21.15.28:0/2681181649 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f28007740 con 0x7f8f300ccd30 2024-09-13T16:45:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.913+0000 7f8f2fe00640 1 -- 172.21.15.28:0/2681181649 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2802d050 con 0x7f8f300ccd30 2024-09-13T16:45:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.913+0000 7f8f2fe00640 1 -- 172.21.15.28:0/2681181649 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f28007ac0 con 0x7f8f300ccd30 2024-09-13T16:45:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.913+0000 7f8f38600640 1 -- 172.21.15.28:0/2681181649 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f300ccd30 msgr2=0x7f8f30155a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.913+0000 7f8f38600640 1 --2- 172.21.15.28:0/2681181649 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f300ccd30 0x7f8f30155a90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8f28009d90 tx=0x7f8f2802a6f0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.914+0000 7f8f38600640 1 -- 172.21.15.28:0/2681181649 shutdown_connections 2024-09-13T16:45:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.914+0000 7f8f38600640 1 --2- 172.21.15.28:0/2681181649 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f300ccd30 0x7f8f30155a90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.914+0000 7f8f38600640 1 --2- 172.21.15.28:0/2681181649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 0x7f8f300c79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.914+0000 7f8f38600640 1 -- 172.21.15.28:0/2681181649 >> 172.21.15.28:0/2681181649 conn(0x7f8f30065c50 msgr2=0x7f8f30066030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.914+0000 7f8f38600640 1 -- 172.21.15.28:0/2681181649 shutdown_connections 2024-09-13T16:45:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.914+0000 7f8f38600640 1 -- 172.21.15.28:0/2681181649 wait complete. 2024-09-13T16:45:18.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.916+0000 7f8f38600640 1 Processor -- start 2024-09-13T16:45:18.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.917+0000 7f8f38600640 1 -- start start 2024-09-13T16:45:18.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.918+0000 7f8f38600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 0x7f8f301ef160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.918+0000 7f8f38600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f300cc950 0x7f8f300c6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.918+0000 7f8f38600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f300ccd30 0x7f8f300c4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.918+0000 7f8f38600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f30159250 con 0x7f8f300cc950 2024-09-13T16:45:18.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.918+0000 7f8f38600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f30158fb0 con 0x7f8f300ccd30 2024-09-13T16:45:18.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.918+0000 7f8f38600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f300c9d50 con 0x7f8f300c7010 2024-09-13T16:45:18.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.919+0000 7f8f35c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 0x7f8f301ef160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.919+0000 7f8f35c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 0x7f8f301ef160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:51580/0 (socket says 172.21.15.28:51580) 2024-09-13T16:45:18.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.919+0000 7f8f35c00640 1 -- 172.21.15.28:0/1002484521 learned_addr learned my addr 172.21.15.28:0/1002484521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:18.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.919+0000 7f8f36600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f300ccd30 0x7f8f300c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.919+0000 7f8f35200640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f300cc950 0x7f8f300c6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.919+0000 7f8f35c00640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 msgr2=0x7f8f301ef160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-13T16:45:18.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.919+0000 7f8f35c00640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 msgr2=0x7f8f301ef160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T16:45:18.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.919+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 0x7f8f301ef160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:18.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.919+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 0x7f8f301ef160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:18.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.919+0000 7f8f36600640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 msgr2=0x7f8f301ef160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:45:18.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.919+0000 7f8f36600640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c7010 0x7f8f301ef160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.919+0000 7f8f36600640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f300cc950 msgr2=0x7f8f300c6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:18.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.920+0000 7f8f36600640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f300cc950 0x7f8f300c6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:18.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.920+0000 7f8f36600640 1 -- 172.21.15.28:0/1002484521 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f28009a70 con 0x7f8f300ccd30 2024-09-13T16:45:18.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.920+0000 7f8f35200640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f300cc950 0x7f8f300c6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:45:18.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.920+0000 7f8f36600640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f300ccd30 0x7f8f300c4200 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8f28031000 tx=0x7f8f28004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:18.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.921+0000 7f8f2d600640 1 -- 172.21.15.28:0/1002484521 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f28031bc0 con 0x7f8f300ccd30 2024-09-13T16:45:18.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.921+0000 7f8f2d600640 1 -- 172.21.15.28:0/1002484521 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f28031d20 con 0x7f8f300ccd30 2024-09-13T16:45:18.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.921+0000 7f8f2d600640 1 -- 172.21.15.28:0/1002484521 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f280456a0 con 0x7f8f300ccd30 2024-09-13T16:45:18.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.921+0000 7f8f38600640 1 -- 172.21.15.28:0/1002484521 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f301ee880 con 0x7f8f300ccd30 2024-09-13T16:45:18.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.921+0000 7f8f38600640 1 -- 172.21.15.28:0/1002484521 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f300c1d20 con 0x7f8f300ccd30 2024-09-13T16:45:18.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.923+0000 7f8f38600640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.923+0000 7f8f38600640 1 -- 172.21.15.28:0/1002484521 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3006ac60 con 0x7f8f300c1eb0 2024-09-13T16:45:18.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.923+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.924+0000 7f8f35c00640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 msgr2=0x7f8f300c5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2024-09-13T16:45:18.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.924+0000 7f8f35c00640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 msgr2=0x7f8f300c5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T16:45:18.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.924+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:18.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.924+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:18.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.924+0000 7f8f2d600640 1 -- 172.21.15.28:0/1002484521 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8f28045800 con 0x7f8f300ccd30 2024-09-13T16:45:18.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.926+0000 7f8f2d600640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8f1006ac00 0x7f8f1006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:18.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.926+0000 7f8f2d600640 1 -- 172.21.15.28:0/1002484521 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8f280aaf70 con 0x7f8f300ccd30 2024-09-13T16:45:18.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.926+0000 7f8f35200640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8f1006ac00 0x7f8f1006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:18.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:18.927+0000 7f8f35200640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8f1006ac00 0x7f8f1006d0c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8f301eb920 tx=0x7f8f1c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:19.126+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:19.126+0000 7f8f35c00640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 msgr2=0x7f8f300c5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-09-13T16:45:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:19.126+0000 7f8f35c00640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 msgr2=0x7f8f300c5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T16:45:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:19.126+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:19.126+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T16:45:19.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:19.527+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:19.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:19.527+0000 7f8f35c00640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 msgr2=0x7f8f300c5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-09-13T16:45:19.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:19.527+0000 7f8f35c00640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 msgr2=0x7f8f300c5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T16:45:19.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:19.527+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:19.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:19.527+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T16:45:20.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:20.328+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:20.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:20.328+0000 7f8f35c00640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 msgr2=0x7f8f300c5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-09-13T16:45:20.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:20.328+0000 7f8f35c00640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 msgr2=0x7f8f300c5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T16:45:20.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:20.328+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:20.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:20.328+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T16:45:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:21.930+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:21.931+0000 7f8f35c00640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 msgr2=0x7f8f300c5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-09-13T16:45:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:21.931+0000 7f8f35c00640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 msgr2=0x7f8f300c5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T16:45:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:21.931+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:21.931+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-13T16:45:22.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.918+0000 7f8f23e00640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 msgr2=0x7f8f300c5300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:45:22.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.919+0000 7f8f23e00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:22.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.919+0000 7f8f23e00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f04003540 0x7f8f040059d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:22.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.919+0000 7f8f23e00640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 msgr2=0x7f8f300c5300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:45:22.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.919+0000 7f8f23e00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c1eb0 0x7f8f300c5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:22.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.919+0000 7f8f23e00640 1 -- 172.21.15.28:0/1002484521 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f04006080 con 0x7f8f04003540 2024-09-13T16:45:22.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.919+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f04003540 0x7f8f040059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:22.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.920+0000 7f8f35c00640 1 -- 172.21.15.28:0/1002484521 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8f040061f0 con 0x7f8f04003540 2024-09-13T16:45:22.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.920+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f04003540 0x7f8f040059d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8f24000f60 tx=0x7f8f24016a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:22.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.920+0000 7f8f2d600640 1 -- 172.21.15.28:0/1002484521 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f24014610 con 0x7f8f04003540 2024-09-13T16:45:22.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.921+0000 7f8f2d600640 1 -- 172.21.15.28:0/1002484521 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f8f2401b070 con 0x7f8f04003540 2024-09-13T16:45:22.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.921+0000 7f8f2d600640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f04003540 msgr2=0x7f8f040059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:22.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.921+0000 7f8f2d600640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f04003540 0x7f8f040059d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8f24000f60 tx=0x7f8f24016a80 comp rx=0 tx=0).stop 2024-09-13T16:45:22.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.921+0000 7f8f2d600640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f04003540 msgr2=0x7f8f040059d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:45:22.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.921+0000 7f8f2d600640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f04003540 0x7f8f040059d0 secure :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8f24000f60 tx=0x7f8f24016a80 comp rx=0 tx=0).stop 2024-09-13T16:45:22.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.934+0000 7f8f38600640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c5840 0x7f8f30150d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:22.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.934+0000 7f8f38600640 1 -- 172.21.15.28:0/1002484521 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f300c5c50 con 0x7f8f300c5840 2024-09-13T16:45:22.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.934+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c5840 0x7f8f30150d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:22.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.935+0000 7f8f35c00640 1 -- 172.21.15.28:0/1002484521 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8f3015cc80 con 0x7f8f300c5840 2024-09-13T16:45:22.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.935+0000 7f8f35c00640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c5840 0x7f8f30150d00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8f24000f60 tx=0x7f8f24000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:22.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.935+0000 7f8f2d600640 1 -- 172.21.15.28:0/1002484521 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f24014820 con 0x7f8f300c5840 2024-09-13T16:45:22.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.936+0000 7f8f2d600640 1 -- 172.21.15.28:0/1002484521 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4457 (secure 0 0 0) 0x7f8f24025580 con 0x7f8f300c5840 2024-09-13T16:45:22.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.936+0000 7f8f2d600640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c5840 msgr2=0x7f8f30150d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:22.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.936+0000 7f8f2d600640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c5840 0x7f8f30150d00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8f24000f60 tx=0x7f8f24000ed0 comp rx=0 tx=0).stop 2024-09-13T16:45:22.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.936+0000 7f8f2d600640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c5840 msgr2=0x7f8f30150d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:45:22.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.936+0000 7f8f2d600640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f300c5840 0x7f8f30150d00 secure :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8f24000f60 tx=0x7f8f24000ed0 comp rx=0 tx=0).stop 2024-09-13T16:45:22.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.936+0000 7f8f36600640 1 -- 172.21.15.28:0/1002484521 reap_dead start 2024-09-13T16:45:22.938 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:45:22.938 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:45:22.938 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:45:22.939 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:45:22.939 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 10, 2024-09-13T16:45:22.939 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:45:22.939 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:45:22.939 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:45:22.939 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:45:22.939 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:22.939 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 4, 2024-09-13T16:45:22.939 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:45:22.939 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:45:22.939 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:45:22.939 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:22.939 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:22.940 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:22.940 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:22.940 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:22.940 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:22.940 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:22.940 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:22.940 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:22.940 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:22.940 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:22.940 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:45:22.940 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:45:22.940 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:22.940 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:22.941 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:22.941 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:22.941 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:22.941 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:22.941 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:22.941 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:22.941 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:22.941 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:22.941 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:22.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:22.941 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:45:22.941 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:45:22.941 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:45:22.941 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:45:22.942 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:45:22.942 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:45:22.942 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:45:22.942 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:45:22.942 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:45:22.942 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:45:22.942 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:45:22.942 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:45:22.942 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:45:22.942 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:45:22.942 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:45:22.942 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:45:22.942 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:45:22.943 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:22.943 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:22.943 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:22.943 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:22.943 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:22.943 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:22.943 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:22.943 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:22.943 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:22.943 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:22.943 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:22.943 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:45:22.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:22.943 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:45:22.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:22.944 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:45:22.944 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:45:22.944 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:22.944 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:22.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:22.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:22.944 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:45:22.944 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:22.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:22.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:22.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:22.944 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:45:22.944 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:22.945 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:22.945 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:22.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:22.945 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:45:22.945 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:45:22.945 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:22.945 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:22.945 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:22.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:22.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:22.946 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:45:22.946 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:45:22.946 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:22.946 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:22.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:22.946 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:22.946 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:45:22.946 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:22.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:22.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:22.946 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:22.946 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:45:22.946 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:22.947 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:22.947 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:22.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:22.947 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:45:22.947 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:45:22.947 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:22.947 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:22.947 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:22.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:22.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:22.947 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:45:22.947 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:45:22.947 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:22.947 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:22.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:22.948 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:22.948 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:45:22.948 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:22.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:22.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:22.948 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:22.948 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:45:22.948 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:22.948 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:22.949 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:22.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:22.949 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:45:22.949 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:45:22.949 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:22.949 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:22.949 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:22.949 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:22.949 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:22.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:22.950 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:45:22.950 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:45:22.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:22.950 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:22.950 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:22.950 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:45:22.950 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:22.950 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:22.950 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:45:22.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:22.950 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:22.950 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:22.951 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:45:22.951 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:22.951 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:22.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:22.951 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:45:22.951 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:45:22.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.936+0000 7f8f38600640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8f1006ac00 msgr2=0x7f8f1006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:22.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.936+0000 7f8f38600640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8f1006ac00 0x7f8f1006d0c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8f301eb920 tx=0x7f8f1c008040 comp rx=0 tx=0).stop 2024-09-13T16:45:22.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.936+0000 7f8f38600640 1 -- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f300ccd30 msgr2=0x7f8f300c4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:22.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.937+0000 7f8f38600640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f300ccd30 0x7f8f300c4200 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8f28031000 tx=0x7f8f28004a40 comp rx=0 tx=0).stop 2024-09-13T16:45:22.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.937+0000 7f8f38600640 1 -- 172.21.15.28:0/1002484521 shutdown_connections 2024-09-13T16:45:22.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.937+0000 7f8f38600640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8f1006ac00 0x7f8f1006d0c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:22.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.937+0000 7f8f38600640 1 --2- 172.21.15.28:0/1002484521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f300ccd30 0x7f8f300c4200 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:22.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.937+0000 7f8f38600640 1 -- 172.21.15.28:0/1002484521 >> 172.21.15.28:0/1002484521 conn(0x7f8f30065c50 msgr2=0x7f8f3006fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:22.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.937+0000 7f8f38600640 1 -- 172.21.15.28:0/1002484521 shutdown_connections 2024-09-13T16:45:22.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:22.937+0000 7f8f38600640 1 -- 172.21.15.28:0/1002484521 wait complete. 2024-09-13T16:45:22.961 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:45:22.962 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-13T16:45:22.962 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-13T16:45:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.134+0000 7f02a4e00640 1 Processor -- start 2024-09-13T16:45:23.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.134+0000 7f02a4e00640 1 -- start start 2024-09-13T16:45:23.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.136+0000 7f02a4e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02a00c9940 0x7f02a00c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:23.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.136+0000 7f02a4e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02a00ca310 con 0x7f02a00ca940 2024-09-13T16:45:23.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.136+0000 7f02a4e00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02a00cbbe0 con 0x7f02a00c46f0 2024-09-13T16:45:23.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.136+0000 7f02a4e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a01532e0 con 0x7f02a00c9940 2024-09-13T16:45:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.137+0000 7f029ea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f02a00ca940 0x7f02a00ce2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52662/0 (socket says 172.21.15.28:52662) 2024-09-13T16:45:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.137+0000 7f029ea00640 1 -- 172.21.15.28:0/899617597 learned_addr learned my addr 172.21.15.28:0/899617597 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:45:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.137+0000 7f029f400640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f02a00c46f0 0x7f02a0159500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:41074/0 (socket says 172.21.15.28:41074) 2024-09-13T16:45:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.137+0000 7f029fe00640 1 --2- 172.21.15.28:0/899617597 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02a00c9940 0x7f02a00c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.137+0000 7f029e000640 1 -- 172.21.15.28:0/899617597 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130287592 0 0) 0x7f02a00cbbe0 con 0x7f02a00c46f0 2024-09-13T16:45:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.137+0000 7f029e000640 1 -- 172.21.15.28:0/899617597 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0284003520 con 0x7f02a00c46f0 2024-09-13T16:45:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.137+0000 7f029fe00640 1 -- 172.21.15.28:0/899617597 >> v1:172.21.15.191:6789/0 conn(0x7f02a00c46f0 legacy=0x7f02a0159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.137+0000 7f029fe00640 1 -- 172.21.15.28:0/899617597 >> v1:172.21.15.28:6789/0 conn(0x7f02a00ca940 legacy=0x7f02a00ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.137+0000 7f029fe00640 1 -- 172.21.15.28:0/899617597 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a00cb060 con 0x7f02a00c9940 2024-09-13T16:45:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.139+0000 7f029fe00640 1 --2- 172.21.15.28:0/899617597 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02a00c9940 0x7f02a00c9dd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0298009dc0 tx=0x7f029802a640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc779259aa78311f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.139+0000 7f029e000640 1 -- 172.21.15.28:0/899617597 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0298007740 con 0x7f02a00c9940 2024-09-13T16:45:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.139+0000 7f029e000640 1 -- 172.21.15.28:0/899617597 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029802d050 con 0x7f02a00c9940 2024-09-13T16:45:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.139+0000 7f029e000640 1 -- 172.21.15.28:0/899617597 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0298007ac0 con 0x7f02a00c9940 2024-09-13T16:45:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.140+0000 7f02a4e00640 1 -- 172.21.15.28:0/899617597 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02a00c9940 msgr2=0x7f02a00c9dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.140+0000 7f02a4e00640 1 --2- 172.21.15.28:0/899617597 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02a00c9940 0x7f02a00c9dd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0298009dc0 tx=0x7f029802a640 comp rx=0 tx=0).stop 2024-09-13T16:45:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.140+0000 7f02a4e00640 1 -- 172.21.15.28:0/899617597 shutdown_connections 2024-09-13T16:45:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.140+0000 7f02a4e00640 1 --2- 172.21.15.28:0/899617597 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02a00c9940 0x7f02a00c9dd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.140+0000 7f02a4e00640 1 -- 172.21.15.28:0/899617597 >> 172.21.15.28:0/899617597 conn(0x7f02a006c830 msgr2=0x7f02a006a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.140+0000 7f02a4e00640 1 -- 172.21.15.28:0/899617597 shutdown_connections 2024-09-13T16:45:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.140+0000 7f02a4e00640 1 -- 172.21.15.28:0/899617597 wait complete. 2024-09-13T16:45:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.142+0000 7f02a4e00640 1 Processor -- start 2024-09-13T16:45:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.142+0000 7f02a4e00640 1 -- start start 2024-09-13T16:45:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.144+0000 7f02a4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02a00c46f0 0x7f02a01593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.144+0000 7f02a4e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02a00c9940 0x7f02a0151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.144+0000 7f02a4e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02a00ca940 0x7f02a01f0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.144+0000 7f02a4e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a0158f60 con 0x7f02a00c46f0 2024-09-13T16:45:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.144+0000 7f02a4e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a00c1c20 con 0x7f02a00ca940 2024-09-13T16:45:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.144+0000 7f02a4e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a00c1940 con 0x7f02a00c9940 2024-09-13T16:45:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.144+0000 7f029fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02a00ca940 0x7f02a01f0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.144+0000 7f029f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02a00c46f0 0x7f02a01593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.144+0000 7f029fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02a00ca940 0x7f02a01f0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:58354/0 (socket says 172.21.15.28:58354) 2024-09-13T16:45:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.144+0000 7f029fe00640 1 -- 172.21.15.28:0/3554154270 learned_addr learned my addr 172.21.15.28:0/3554154270 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.144+0000 7f029ea00640 1 --2- 172.21.15.28:0/3554154270 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02a00c9940 0x7f02a0151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.145+0000 7f029ea00640 1 -- 172.21.15.28:0/3554154270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02a00ca940 msgr2=0x7f02a01f0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.145+0000 7f029ea00640 1 --2- 172.21.15.28:0/3554154270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02a00ca940 0x7f02a01f0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.145+0000 7f029ea00640 1 -- 172.21.15.28:0/3554154270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02a00c46f0 msgr2=0x7f02a01593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.145+0000 7f029ea00640 1 --2- 172.21.15.28:0/3554154270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02a00c46f0 0x7f02a01593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.145+0000 7f029ea00640 1 -- 172.21.15.28:0/3554154270 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0298009a70 con 0x7f02a00c9940 2024-09-13T16:45:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.145+0000 7f029fe00640 1 --2- 172.21.15.28:0/3554154270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02a00ca940 0x7f02a01f0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:45:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.145+0000 7f029f400640 1 --2- 172.21.15.28:0/3554154270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02a00c46f0 0x7f02a01593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:45:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.145+0000 7f029ea00640 1 --2- 172.21.15.28:0/3554154270 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02a00c9940 0x7f02a0151e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f028c00c440 tx=0x7f028c00c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.145+0000 7f0297400640 1 -- 172.21.15.28:0/3554154270 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c005ce0 con 0x7f02a00c9940 2024-09-13T16:45:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.145+0000 7f0297400640 1 -- 172.21.15.28:0/3554154270 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028c005e40 con 0x7f02a00c9940 2024-09-13T16:45:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.145+0000 7f0297400640 1 -- 172.21.15.28:0/3554154270 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c0072f0 con 0x7f02a00c9940 2024-09-13T16:45:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.145+0000 7f02a4e00640 1 -- 172.21.15.28:0/3554154270 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a0158630 con 0x7f02a00c9940 2024-09-13T16:45:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.146+0000 7f02a4e00640 1 -- 172.21.15.28:0/3554154270 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a0158bf0 con 0x7f02a00c9940 2024-09-13T16:45:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.147+0000 7f02a4e00640 1 -- 172.21.15.28:0/3554154270 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a0067460 con 0x7f02a00c9940 2024-09-13T16:45:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.147+0000 7f0297400640 1 -- 172.21.15.28:0/3554154270 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f028c007450 con 0x7f02a00c9940 2024-09-13T16:45:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.148+0000 7f0297400640 1 --2- 172.21.15.28:0/3554154270 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f028006ac00 0x7f028006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.148+0000 7f0297400640 1 -- 172.21.15.28:0/3554154270 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f028c088900 con 0x7f02a00c9940 2024-09-13T16:45:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.149+0000 7f029f400640 1 --2- 172.21.15.28:0/3554154270 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f028006ac00 0x7f028006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.149+0000 7f029f400640 1 --2- 172.21.15.28:0/3554154270 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f028006ac00 0x7f028006d0c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f02900011e0 tx=0x7f0290007d30 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.153+0000 7f0297400640 1 -- 172.21.15.28:0/3554154270 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f028c053a70 con 0x7f02a00c9940 2024-09-13T16:45:23.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.652+0000 7f02a4e00640 1 -- 172.21.15.28:0/3554154270 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f02a00b4110 con 0x7f02a00c9940 2024-09-13T16:45:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.654+0000 7f0297400640 1 -- 172.21.15.28:0/3554154270 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f028c017080 con 0x7f02a00c9940 2024-09-13T16:45:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.658+0000 7f02a4e00640 1 -- 172.21.15.28:0/3554154270 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f028006ac00 msgr2=0x7f028006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.658+0000 7f02a4e00640 1 --2- 172.21.15.28:0/3554154270 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f028006ac00 0x7f028006d0c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f02900011e0 tx=0x7f0290007d30 comp rx=0 tx=0).stop 2024-09-13T16:45:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.658+0000 7f02a4e00640 1 -- 172.21.15.28:0/3554154270 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02a00c9940 msgr2=0x7f02a0151e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.658+0000 7f02a4e00640 1 --2- 172.21.15.28:0/3554154270 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02a00c9940 0x7f02a0151e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f028c00c440 tx=0x7f028c00c900 comp rx=0 tx=0).stop 2024-09-13T16:45:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.658+0000 7f02a4e00640 1 -- 172.21.15.28:0/3554154270 shutdown_connections 2024-09-13T16:45:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.658+0000 7f02a4e00640 1 --2- 172.21.15.28:0/3554154270 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f028006ac00 0x7f028006d0c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.658+0000 7f02a4e00640 1 --2- 172.21.15.28:0/3554154270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02a00ca940 0x7f02a01f0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.658+0000 7f02a4e00640 1 --2- 172.21.15.28:0/3554154270 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02a00c9940 0x7f02a0151e20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.658+0000 7f02a4e00640 1 --2- 172.21.15.28:0/3554154270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02a00c46f0 0x7f02a01593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.658+0000 7f02a4e00640 1 -- 172.21.15.28:0/3554154270 >> 172.21.15.28:0/3554154270 conn(0x7f02a006c830 msgr2=0x7f02a0155450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.658+0000 7f02a4e00640 1 -- 172.21.15.28:0/3554154270 shutdown_connections 2024-09-13T16:45:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:23.658+0000 7f02a4e00640 1 -- 172.21.15.28:0/3554154270 wait complete. 2024-09-13T16:45:23.684 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-09-13T16:45:33.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:45:33.588 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:45:33.685 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T16:45:33.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:45:33.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.848+0000 7fe867200640 1 Processor -- start 2024-09-13T16:45:33.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.848+0000 7fe867200640 1 -- start start 2024-09-13T16:45:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.849+0000 7fe867200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe8680c5060 0x7fe8680c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.849+0000 7fe867200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8680c5440 0x7fe8680c2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.849+0000 7fe867200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe868156610 0x7fe868150400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.849+0000 7fe867200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8680c5df0 con 0x7fe8680c5060 2024-09-13T16:45:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.849+0000 7fe867200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8680c2840 con 0x7fe868156610 2024-09-13T16:45:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.849+0000 7fe867200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8680c2ab0 con 0x7fe8680c5440 2024-09-13T16:45:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.850+0000 7fe865e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe8680c5060 0x7fe8680c44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.850+0000 7fe865e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe8680c5060 0x7fe8680c44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59128/0 (socket says 172.21.15.28:59128) 2024-09-13T16:45:33.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.850+0000 7fe865e00640 1 -- 172.21.15.28:0/674695455 learned_addr learned my addr 172.21.15.28:0/674695455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:33.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.850+0000 7fe865400640 1 --2- 172.21.15.28:0/674695455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8680c5440 0x7fe8680c2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:33.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.850+0000 7fe866800640 1 --2- 172.21.15.28:0/674695455 >> v2:172.21.15.191:3300/0 conn(0x7fe868156610 0x7fe868150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.850+0000 7fe865400640 1 -- 172.21.15.28:0/674695455 >> v2:172.21.15.191:3300/0 conn(0x7fe868156610 msgr2=0x7fe868150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:45:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.850+0000 7fe865400640 1 --2- 172.21.15.28:0/674695455 >> v2:172.21.15.191:3300/0 conn(0x7fe868156610 0x7fe868150400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.850+0000 7fe865400640 1 -- 172.21.15.28:0/674695455 >> v2:172.21.15.28:3300/0 conn(0x7fe8680c5060 msgr2=0x7fe8680c44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:45:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.850+0000 7fe865400640 1 --2- 172.21.15.28:0/674695455 >> v2:172.21.15.28:3300/0 conn(0x7fe8680c5060 0x7fe8680c44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.850+0000 7fe865400640 1 -- 172.21.15.28:0/674695455 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8680c2d70 con 0x7fe8680c5440 2024-09-13T16:45:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.850+0000 7fe865e00640 1 --2- 172.21.15.28:0/674695455 >> v2:172.21.15.28:3300/0 conn(0x7fe8680c5060 0x7fe8680c44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:45:33.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.852+0000 7fe865400640 1 --2- 172.21.15.28:0/674695455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8680c5440 0x7fe8680c2300 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe86806ad90 tx=0x7fe85402a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e149baefd9fc50f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:33.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.852+0000 7fe864a00640 1 -- 172.21.15.28:0/674695455 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe854032440 con 0x7fe8680c5440 2024-09-13T16:45:33.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.852+0000 7fe864a00640 1 -- 172.21.15.28:0/674695455 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8540325a0 con 0x7fe8680c5440 2024-09-13T16:45:33.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.852+0000 7fe864a00640 1 -- 172.21.15.28:0/674695455 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe854032930 con 0x7fe8680c5440 2024-09-13T16:45:33.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.852+0000 7fe867200640 1 -- 172.21.15.28:0/674695455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8680c5440 msgr2=0x7fe8680c2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:33.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.852+0000 7fe867200640 1 --2- 172.21.15.28:0/674695455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8680c5440 0x7fe8680c2300 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe86806ad90 tx=0x7fe85402a1a0 comp rx=0 tx=0).stop 2024-09-13T16:45:33.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.853+0000 7fe867200640 1 -- 172.21.15.28:0/674695455 shutdown_connections 2024-09-13T16:45:33.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.853+0000 7fe867200640 1 --2- 172.21.15.28:0/674695455 >> v2:172.21.15.191:3300/0 conn(0x7fe868156610 0x7fe868150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:33.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.853+0000 7fe867200640 1 --2- 172.21.15.28:0/674695455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8680c5440 0x7fe8680c2300 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:33.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.853+0000 7fe867200640 1 --2- 172.21.15.28:0/674695455 >> v2:172.21.15.28:3300/0 conn(0x7fe8680c5060 0x7fe8680c44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:33.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.853+0000 7fe867200640 1 -- 172.21.15.28:0/674695455 >> 172.21.15.28:0/674695455 conn(0x7fe868069e80 msgr2=0x7fe86806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:33.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.853+0000 7fe867200640 1 -- 172.21.15.28:0/674695455 shutdown_connections 2024-09-13T16:45:33.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.853+0000 7fe867200640 1 -- 172.21.15.28:0/674695455 wait complete. 2024-09-13T16:45:33.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.854+0000 7fe867200640 1 Processor -- start 2024-09-13T16:45:33.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.854+0000 7fe867200640 1 -- start start 2024-09-13T16:45:33.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.855+0000 7fe867200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8680c5060 0x7fe868152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:33.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.855+0000 7fe867200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8680c5440 0x7fe868153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:33.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.855+0000 7fe867200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe868156610 0x7fe868157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:33.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.855+0000 7fe867200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe868159410 con 0x7fe8680c5060 2024-09-13T16:45:33.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.855+0000 7fe867200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe868157320 con 0x7fe868156610 2024-09-13T16:45:33.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.855+0000 7fe867200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe868153a90 con 0x7fe8680c5440 2024-09-13T16:45:33.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.856+0000 7fe865e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8680c5060 0x7fe868152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:33.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.856+0000 7fe865e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8680c5060 0x7fe868152d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59138/0 (socket says 172.21.15.28:59138) 2024-09-13T16:45:33.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.856+0000 7fe865e00640 1 -- 172.21.15.28:0/2368129695 learned_addr learned my addr 172.21.15.28:0/2368129695 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:33.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.856+0000 7fe865400640 1 --2- 172.21.15.28:0/2368129695 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8680c5440 0x7fe868153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:33.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.856+0000 7fe866800640 1 --2- 172.21.15.28:0/2368129695 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe868156610 0x7fe868157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:33.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.856+0000 7fe865e00640 1 -- 172.21.15.28:0/2368129695 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8680c5440 msgr2=0x7fe868153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.856+0000 7fe865e00640 1 --2- 172.21.15.28:0/2368129695 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8680c5440 0x7fe868153240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.856+0000 7fe865e00640 1 -- 172.21.15.28:0/2368129695 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe868156610 msgr2=0x7fe868157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.856+0000 7fe865e00640 1 --2- 172.21.15.28:0/2368129695 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe868156610 0x7fe868157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.856+0000 7fe865e00640 1 -- 172.21.15.28:0/2368129695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8580094d0 con 0x7fe8680c5060 2024-09-13T16:45:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.856+0000 7fe865e00640 1 --2- 172.21.15.28:0/2368129695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8680c5060 0x7fe868152d00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe85800e8d0 tx=0x7fe85800ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.857+0000 7fe85e000640 1 -- 172.21.15.28:0/2368129695 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe858004030 con 0x7fe8680c5060 2024-09-13T16:45:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.857+0000 7fe85e000640 1 -- 172.21.15.28:0/2368129695 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe858004190 con 0x7fe8680c5060 2024-09-13T16:45:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.857+0000 7fe85e000640 1 -- 172.21.15.28:0/2368129695 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe858004500 con 0x7fe8680c5060 2024-09-13T16:45:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.857+0000 7fe867200640 1 -- 172.21.15.28:0/2368129695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe854009400 con 0x7fe8680c5060 2024-09-13T16:45:33.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.857+0000 7fe867200640 1 -- 172.21.15.28:0/2368129695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8681eefe0 con 0x7fe8680c5060 2024-09-13T16:45:33.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.858+0000 7fe867200640 1 -- 172.21.15.28:0/2368129695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8680c7d00 con 0x7fe8680c5060 2024-09-13T16:45:33.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.859+0000 7fe85e000640 1 -- 172.21.15.28:0/2368129695 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe85800c940 con 0x7fe8680c5060 2024-09-13T16:45:33.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.861+0000 7fe85e000640 1 --2- 172.21.15.28:0/2368129695 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe84806ac80 0x7fe84806d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:33.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.861+0000 7fe85e000640 1 -- 172.21.15.28:0/2368129695 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe858089150 con 0x7fe8680c5060 2024-09-13T16:45:33.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.861+0000 7fe865400640 1 --2- 172.21.15.28:0/2368129695 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe84806ac80 0x7fe84806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:33.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.862+0000 7fe865400640 1 --2- 172.21.15.28:0/2368129695 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe84806ac80 0x7fe84806d140 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe854034000 tx=0x7fe85402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:33.870+0000 7fe85e000640 1 -- 172.21.15.28:0/2368129695 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe8580544d0 con 0x7fe8680c5060 2024-09-13T16:45:34.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.165+0000 7fe867200640 1 -- 172.21.15.28:0/2368129695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe868157590 con 0x7fe8680c5060 2024-09-13T16:45:34.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.165+0000 7fe85e000640 1 -- 172.21.15.28:0/2368129695 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7fe868157590 con 0x7fe8680c5060 2024-09-13T16:45:34.167 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:45:34.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.167+0000 7fe867200640 1 -- 172.21.15.28:0/2368129695 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe84806ac80 msgr2=0x7fe84806d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.167+0000 7fe867200640 1 --2- 172.21.15.28:0/2368129695 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe84806ac80 0x7fe84806d140 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe854034000 tx=0x7fe85402e000 comp rx=0 tx=0).stop 2024-09-13T16:45:34.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.167+0000 7fe867200640 1 -- 172.21.15.28:0/2368129695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8680c5060 msgr2=0x7fe868152d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.167+0000 7fe867200640 1 --2- 172.21.15.28:0/2368129695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8680c5060 0x7fe868152d00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe85800e8d0 tx=0x7fe85800ed90 comp rx=0 tx=0).stop 2024-09-13T16:45:34.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.167+0000 7fe867200640 1 -- 172.21.15.28:0/2368129695 shutdown_connections 2024-09-13T16:45:34.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.167+0000 7fe867200640 1 --2- 172.21.15.28:0/2368129695 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe84806ac80 0x7fe84806d140 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.167+0000 7fe867200640 1 --2- 172.21.15.28:0/2368129695 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe868156610 0x7fe868157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.167+0000 7fe867200640 1 --2- 172.21.15.28:0/2368129695 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8680c5440 0x7fe868153240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.167+0000 7fe867200640 1 --2- 172.21.15.28:0/2368129695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8680c5060 0x7fe868152d00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.168+0000 7fe867200640 1 -- 172.21.15.28:0/2368129695 >> 172.21.15.28:0/2368129695 conn(0x7fe868069e80 msgr2=0x7fe86814d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:34.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.168+0000 7fe867200640 1 -- 172.21.15.28:0/2368129695 shutdown_connections 2024-09-13T16:45:34.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.168+0000 7fe867200640 1 -- 172.21.15.28:0/2368129695 wait complete. 2024-09-13T16:45:34.185 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T16:45:34.186 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-13T16:45:34.186 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-13T16:45:34.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T16:45:34.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.345+0000 7fd93aa00640 1 Processor -- start 2024-09-13T16:45:34.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.345+0000 7fd93aa00640 1 -- start start 2024-09-13T16:45:34.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.346+0000 7fd93aa00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd93415a440 0x7fd9340c5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.346+0000 7fd93aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd93406dcb0 0x7fd9341504e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.346+0000 7fd93aa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9340c1d80 con 0x7fd93415a060 2024-09-13T16:45:34.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.346+0000 7fd93aa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd93415a920 con 0x7fd93406dcb0 2024-09-13T16:45:34.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.346+0000 7fd93aa00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd93415ac00 con 0x7fd93415a440 2024-09-13T16:45:34.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.347+0000 7fd933400640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd93415a440 0x7fd9340c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.347+0000 7fd933400640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd93415a440 0x7fd9340c5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:44606/0 (socket says 172.21.15.28:44606) 2024-09-13T16:45:34.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.347+0000 7fd933400640 1 -- 172.21.15.28:0/2972226865 learned_addr learned my addr 172.21.15.28:0/2972226865 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:34.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.347+0000 7fd933400640 1 -- 172.21.15.28:0/2972226865 >> v2:172.21.15.28:3300/0 conn(0x7fd93406dcb0 msgr2=0x7fd9341504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:45:34.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.347+0000 7fd933400640 1 --2- 172.21.15.28:0/2972226865 >> v2:172.21.15.28:3300/0 conn(0x7fd93406dcb0 0x7fd9341504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.347+0000 7fd933400640 1 -- 172.21.15.28:0/2972226865 >> v1:172.21.15.28:6789/0 conn(0x7fd93415a060 legacy=0x7fd93406d5a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:45:34.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.347+0000 7fd933400640 1 -- 172.21.15.28:0/2972226865 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9340c2d50 con 0x7fd93415a440 2024-09-13T16:45:34.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.348+0000 7fd933400640 1 --2- 172.21.15.28:0/2972226865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93415a440 0x7fd9340c5500 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd9340c47e0 tx=0x7fd92002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28bf6bfc3b97d7f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:34.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.349+0000 7fd932a00640 1 -- 172.21.15.28:0/2972226865 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd920032440 con 0x7fd93415a440 2024-09-13T16:45:34.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.349+0000 7fd932a00640 1 -- 172.21.15.28:0/2972226865 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9200325a0 con 0x7fd93415a440 2024-09-13T16:45:34.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.349+0000 7fd932a00640 1 -- 172.21.15.28:0/2972226865 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd920032930 con 0x7fd93415a440 2024-09-13T16:45:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.349+0000 7fd93aa00640 1 -- 172.21.15.28:0/2972226865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93415a440 msgr2=0x7fd9340c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.349+0000 7fd93aa00640 1 --2- 172.21.15.28:0/2972226865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93415a440 0x7fd9340c5500 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd9340c47e0 tx=0x7fd92002a230 comp rx=0 tx=0).stop 2024-09-13T16:45:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.350+0000 7fd93aa00640 1 -- 172.21.15.28:0/2972226865 shutdown_connections 2024-09-13T16:45:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.350+0000 7fd93aa00640 1 --2- 172.21.15.28:0/2972226865 >> v2:172.21.15.28:3300/0 conn(0x7fd93406dcb0 0x7fd9341504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.350+0000 7fd93aa00640 1 --2- 172.21.15.28:0/2972226865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93415a440 0x7fd9340c5500 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.350+0000 7fd93aa00640 1 -- 172.21.15.28:0/2972226865 >> 172.21.15.28:0/2972226865 conn(0x7fd9340ca6b0 msgr2=0x7fd9340caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:34.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.350+0000 7fd93aa00640 1 -- 172.21.15.28:0/2972226865 shutdown_connections 2024-09-13T16:45:34.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.350+0000 7fd93aa00640 1 -- 172.21.15.28:0/2972226865 wait complete. 2024-09-13T16:45:34.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.351+0000 7fd93aa00640 1 Processor -- start 2024-09-13T16:45:34.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.352+0000 7fd93aa00640 1 -- start start 2024-09-13T16:45:34.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.352+0000 7fd93aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93406dcb0 0x7fd934155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.352+0000 7fd93aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93415a060 0x7fd9340c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.353+0000 7fd93aa00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd93415a440 0x7fd934070e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.353+0000 7fd93aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd934154020 con 0x7fd93415a060 2024-09-13T16:45:34.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.353+0000 7fd93aa00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd934155e70 con 0x7fd93406dcb0 2024-09-13T16:45:34.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.353+0000 7fd93aa00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd934153a60 con 0x7fd93415a440 2024-09-13T16:45:34.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.353+0000 7fd933400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93415a060 0x7fd9340c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.353+0000 7fd933e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93406dcb0 0x7fd934155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.353+0000 7fd933400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93415a060 0x7fd9340c8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59156/0 (socket says 172.21.15.28:59156) 2024-09-13T16:45:34.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.353+0000 7fd933400640 1 -- 172.21.15.28:0/926481562 learned_addr learned my addr 172.21.15.28:0/926481562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:34.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.353+0000 7fd938a00640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd93415a440 0x7fd934070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.354+0000 7fd933e00640 1 -- 172.21.15.28:0/926481562 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd93415a440 msgr2=0x7fd934070e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.354+0000 7fd933e00640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd93415a440 0x7fd934070e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.354+0000 7fd933e00640 1 -- 172.21.15.28:0/926481562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93415a060 msgr2=0x7fd9340c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.354+0000 7fd933e00640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93415a060 0x7fd9340c8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.354+0000 7fd933e00640 1 -- 172.21.15.28:0/926481562 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9200094d0 con 0x7fd93406dcb0 2024-09-13T16:45:34.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.354+0000 7fd938a00640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd93415a440 0x7fd934070e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:45:34.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.354+0000 7fd933400640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93415a060 0x7fd9340c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:45:34.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.354+0000 7fd933e00640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93406dcb0 0x7fd934155930 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd92400c410 tx=0x7fd92400c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:34.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.354+0000 7fd92be00640 1 -- 172.21.15.28:0/926481562 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd924005230 con 0x7fd93406dcb0 2024-09-13T16:45:34.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.354+0000 7fd92be00640 1 -- 172.21.15.28:0/926481562 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd924005390 con 0x7fd93406dcb0 2024-09-13T16:45:34.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.355+0000 7fd92be00640 1 -- 172.21.15.28:0/926481562 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd924004820 con 0x7fd93406dcb0 2024-09-13T16:45:34.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.355+0000 7fd93aa00640 1 -- 172.21.15.28:0/926481562 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd934153d40 con 0x7fd93406dcb0 2024-09-13T16:45:34.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.355+0000 7fd93aa00640 1 -- 172.21.15.28:0/926481562 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9341eeb60 con 0x7fd93406dcb0 2024-09-13T16:45:34.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.356+0000 7fd93aa00640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9341eed50 0x7fd9340713d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.356+0000 7fd93aa00640 1 -- 172.21.15.28:0/926481562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd934066c20 con 0x7fd9341eed50 2024-09-13T16:45:34.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.357+0000 7fd933400640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9341eed50 0x7fd9340713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.357+0000 7fd933400640 1 -- 172.21.15.28:0/926481562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd9340c5220 con 0x7fd9341eed50 2024-09-13T16:45:34.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.358+0000 7fd933400640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9341eed50 0x7fd9340713d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd920035040 tx=0x7fd9200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:34.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.358+0000 7fd92be00640 1 -- 172.21.15.28:0/926481562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92003be60 con 0x7fd9341eed50 2024-09-13T16:45:34.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.358+0000 7fd92be00640 1 -- 172.21.15.28:0/926481562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fd920040070 con 0x7fd9341eed50 2024-09-13T16:45:34.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.358+0000 7fd92be00640 1 -- 172.21.15.28:0/926481562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9341eed50 msgr2=0x7fd9340713d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.358+0000 7fd92be00640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9341eed50 0x7fd9340713d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd920035040 tx=0x7fd9200046c0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.358+0000 7fd92be00640 1 -- 172.21.15.28:0/926481562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9341eed50 msgr2=0x7fd9340713d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:45:34.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.358+0000 7fd92be00640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9341eed50 0x7fd9340713d0 secure :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd920035040 tx=0x7fd9200046c0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.358+0000 7fd92be00640 1 -- 172.21.15.28:0/926481562 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd924004980 con 0x7fd93406dcb0 2024-09-13T16:45:34.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.360+0000 7fd92be00640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd91406a8c0 0x7fd91406cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.360+0000 7fd92be00640 1 -- 172.21.15.28:0/926481562 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd924088900 con 0x7fd93406dcb0 2024-09-13T16:45:34.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.360+0000 7fd933400640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd91406a8c0 0x7fd91406cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.361+0000 7fd933400640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd91406a8c0 0x7fd91406cd80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd920044000 tx=0x7fd920042000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:34.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.365+0000 7fd93aa00640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9341ef340 0x7fd9341f1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.366+0000 7fd93aa00640 1 -- 172.21.15.28:0/926481562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd934071910 con 0x7fd9341ef340 2024-09-13T16:45:34.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.366+0000 7fd938a00640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9341ef340 0x7fd9341f1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.366+0000 7fd938a00640 1 -- 172.21.15.28:0/926481562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd9340c3310 con 0x7fd9341ef340 2024-09-13T16:45:34.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.367+0000 7fd938a00640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9341ef340 0x7fd9341f1c50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00e2f0 tx=0x7fd92c00e6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:34.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.367+0000 7fd92be00640 1 -- 172.21.15.28:0/926481562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c010040 con 0x7fd9341ef340 2024-09-13T16:45:34.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.367+0000 7fd92be00640 1 -- 172.21.15.28:0/926481562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+5101 (secure 0 0 0) 0x7fd92c014610 con 0x7fd9341ef340 2024-09-13T16:45:34.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.367+0000 7fd92be00640 1 -- 172.21.15.28:0/926481562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9341ef340 msgr2=0x7fd9341f1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.367+0000 7fd92be00640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9341ef340 0x7fd9341f1c50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00e2f0 tx=0x7fd92c00e6f0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.367+0000 7fd92be00640 1 -- 172.21.15.28:0/926481562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9341ef340 msgr2=0x7fd9341f1c50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:45:34.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.367+0000 7fd92be00640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9341ef340 0x7fd9341f1c50 secure :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00e2f0 tx=0x7fd92c00e6f0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.370 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:45:34.370 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:45:34.370 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:45:34.370 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "leader", 2024-09-13T16:45:34.370 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 10, 2024-09-13T16:45:34.370 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:45:34.370 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:45:34.370 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:45:34.370 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:45:34.370 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.370 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 16, 2024-09-13T16:45:34.371 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:45:34.371 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:45:34.371 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:45:34.371 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:34.371 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:34.371 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:34.371 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:34.371 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:34.371 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:34.371 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:34.371 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:34.371 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:34.372 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:34.372 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.372 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:45:34.372 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:45:34.372 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:34.372 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:34.372 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:34.372 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:34.372 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:34.372 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:34.372 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:34.372 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:34.372 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:34.373 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:34.373 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.373 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:45:34.373 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:45:34.373 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:45:34.373 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:45:34.373 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:45:34.373 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:45:34.373 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:45:34.373 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:45:34.373 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:45:34.373 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:45:34.374 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:45:34.374 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:45:34.374 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:45:34.374 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:45:34.374 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:45:34.374 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:45:34.374 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:45:34.374 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:34.374 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:34.374 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:34.374 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:34.374 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:34.374 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:34.374 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:34.375 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:34.375 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:34.375 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:34.375 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.375 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:45:34.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.375 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:45:34.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.375 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:45:34.375 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:45:34.375 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:34.375 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:34.375 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:34.376 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:45:34.376 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:34.376 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:45:34.376 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.376 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.376 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.376 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:45:34.377 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:45:34.377 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:34.377 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:34.377 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:34.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.377 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:45:34.377 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:45:34.377 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:34.377 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:34.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:34.378 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:45:34.378 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.378 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:34.378 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:45:34.378 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.378 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.378 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.378 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:45:34.379 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:45:34.379 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:34.379 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:34.379 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:34.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.379 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:45:34.379 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:45:34.379 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:34.379 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:34.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:34.380 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:45:34.380 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.380 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.380 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:34.380 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:45:34.380 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.380 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.380 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.380 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:45:34.380 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:45:34.380 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:34.381 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:34.381 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:34.381 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.381 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.381 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:45:34.381 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:45:34.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.381 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:34.381 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:34.381 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:45:34.381 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.382 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.382 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:45:34.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.382 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:34.382 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:34.382 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:45:34.382 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.382 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.382 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:45:34.382 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.382 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:34.382 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:34.382 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4 2024-09-13T16:45:34.383 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.383 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.383 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:45:34.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.383 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x2f018fb87aa4aafe", 2024-09-13T16:45:34.383 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "luminous", 2024-09-13T16:45:34.383 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:45:34.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.383 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:34.383 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:34.383 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:45:34.384 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.384 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.384 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": [ 2024-09-13T16:45:34.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.384 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:34.384 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:34.384 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:45:34.384 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.384 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.384 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:45:34.384 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:45:34.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.368+0000 7fd93aa00640 1 -- 172.21.15.28:0/926481562 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd91406a8c0 msgr2=0x7fd91406cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.368+0000 7fd93aa00640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd91406a8c0 0x7fd91406cd80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd920044000 tx=0x7fd920042000 comp rx=0 tx=0).stop 2024-09-13T16:45:34.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.368+0000 7fd93aa00640 1 -- 172.21.15.28:0/926481562 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93406dcb0 msgr2=0x7fd934155930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.368+0000 7fd93aa00640 1 --2- 172.21.15.28:0/926481562 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93406dcb0 0x7fd934155930 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd92400c410 tx=0x7fd92400c8d0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.368+0000 7fd938a00640 1 -- 172.21.15.28:0/926481562 reap_dead start 2024-09-13T16:45:34.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.368+0000 7fd93aa00640 1 -- 172.21.15.28:0/926481562 shutdown_connections 2024-09-13T16:45:34.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.368+0000 7fd93aa00640 1 -- 172.21.15.28:0/926481562 >> 172.21.15.28:0/926481562 conn(0x7fd9340ca6b0 msgr2=0x7fd9340c1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:34.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.368+0000 7fd93aa00640 1 -- 172.21.15.28:0/926481562 shutdown_connections 2024-09-13T16:45:34.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.369+0000 7fd93aa00640 1 -- 172.21.15.28:0/926481562 wait complete. 2024-09-13T16:45:34.387 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:45:34.387 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-13T16:45:34.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T16:45:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.548+0000 7f56b9e00640 1 Processor -- start 2024-09-13T16:45:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.548+0000 7f56b9e00640 1 -- start start 2024-09-13T16:45:34.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.549+0000 7f56b9e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f56b4152da0 0x7f56b40cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.550+0000 7f56b9e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40cadd0 0x7f56b4158b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.550+0000 7f56b9e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b40ce7c0 con 0x7f56b40c9500 2024-09-13T16:45:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.550+0000 7f56b9e00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b40c5060 con 0x7f56b4152da0 2024-09-13T16:45:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.550+0000 7f56b9e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40cc260 con 0x7f56b40cadd0 2024-09-13T16:45:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.550+0000 7f56b8a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f56b4152da0 0x7f56b40cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.550+0000 7f56b8a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f56b4152da0 0x7f56b40cbd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:44624/0 (socket says 172.21.15.28:44624) 2024-09-13T16:45:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.550+0000 7f56b8a00640 1 -- 172.21.15.28:0/3164278173 learned_addr learned my addr 172.21.15.28:0/3164278173 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.550+0000 7f56b3e00640 1 --2- 172.21.15.28:0/3164278173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40cadd0 0x7f56b4158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.550+0000 7f56b8a00640 1 -- 172.21.15.28:0/3164278173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40cadd0 msgr2=0x7f56b4158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:45:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.550+0000 7f56b8a00640 1 --2- 172.21.15.28:0/3164278173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40cadd0 0x7f56b4158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.550+0000 7f56b8a00640 1 -- 172.21.15.28:0/3164278173 >> v1:172.21.15.28:6789/0 conn(0x7f56b40c9500 legacy=0x7f56b406aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.550+0000 7f56b8a00640 1 -- 172.21.15.28:0/3164278173 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b40cf060 con 0x7f56b4152da0 2024-09-13T16:45:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.552+0000 7f56b8a00640 1 --2- 172.21.15.28:0/3164278173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b4152da0 0x7f56b40cbd20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f56a8009870 tx=0x7f56a802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=987a091d05f2e154 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.552+0000 7f56b3400640 1 -- 172.21.15.28:0/3164278173 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a8032440 con 0x7f56b4152da0 2024-09-13T16:45:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.552+0000 7f56b3400640 1 -- 172.21.15.28:0/3164278173 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a80325a0 con 0x7f56b4152da0 2024-09-13T16:45:34.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.552+0000 7f56b3400640 1 -- 172.21.15.28:0/3164278173 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a8032930 con 0x7f56b4152da0 2024-09-13T16:45:34.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.553+0000 7f56b9e00640 1 -- 172.21.15.28:0/3164278173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b4152da0 msgr2=0x7f56b40cbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.553+0000 7f56b9e00640 1 --2- 172.21.15.28:0/3164278173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b4152da0 0x7f56b40cbd20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f56a8009870 tx=0x7f56a802a250 comp rx=0 tx=0).stop 2024-09-13T16:45:34.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.553+0000 7f56b9e00640 1 -- 172.21.15.28:0/3164278173 shutdown_connections 2024-09-13T16:45:34.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.553+0000 7f56b9e00640 1 --2- 172.21.15.28:0/3164278173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b40cadd0 0x7f56b4158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.553+0000 7f56b9e00640 1 --2- 172.21.15.28:0/3164278173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b4152da0 0x7f56b40cbd20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.553+0000 7f56b9e00640 1 -- 172.21.15.28:0/3164278173 >> 172.21.15.28:0/3164278173 conn(0x7f56b406c180 msgr2=0x7f56b40715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:34.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.553+0000 7f56b9e00640 1 -- 172.21.15.28:0/3164278173 shutdown_connections 2024-09-13T16:45:34.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.553+0000 7f56b9e00640 1 -- 172.21.15.28:0/3164278173 wait complete. 2024-09-13T16:45:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.555+0000 7f56b9e00640 1 Processor -- start 2024-09-13T16:45:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.555+0000 7f56b9e00640 1 -- start start 2024-09-13T16:45:34.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.556+0000 7f56b9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56b40c9500 0x7f56b414eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.556+0000 7f56b9e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b40cadd0 0x7f56b4145130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.556+0000 7f56b9e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b4152da0 0x7f56b4145670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.556+0000 7f56b9e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40c8530 con 0x7f56b40c9500 2024-09-13T16:45:34.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.556+0000 7f56b9e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40c5bc0 con 0x7f56b40cadd0 2024-09-13T16:45:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.556+0000 7f56b9e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b41f21c0 con 0x7f56b4152da0 2024-09-13T16:45:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.556+0000 7f56b8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56b40c9500 0x7f56b414eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.556+0000 7f56b3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b40cadd0 0x7f56b4145130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.556+0000 7f56b8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56b40c9500 0x7f56b414eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59176/0 (socket says 172.21.15.28:59176) 2024-09-13T16:45:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.556+0000 7f56b3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b40cadd0 0x7f56b4145130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:44630/0 (socket says 172.21.15.28:44630) 2024-09-13T16:45:34.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.556+0000 7f56b8a00640 1 -- 172.21.15.28:0/290208438 learned_addr learned my addr 172.21.15.28:0/290208438 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:34.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.557+0000 7f56b9400640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b4152da0 0x7f56b4145670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.557+0000 7f56b8a00640 1 -- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b4152da0 msgr2=0x7f56b4145670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.557+0000 7f56b8a00640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56b4152da0 0x7f56b4145670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.557+0000 7f56b8a00640 1 -- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b40cadd0 msgr2=0x7f56b4145130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.557+0000 7f56b8a00640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b40cadd0 0x7f56b4145130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.557+0000 7f56b8a00640 1 -- 172.21.15.28:0/290208438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a80094d0 con 0x7f56b40c9500 2024-09-13T16:45:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.557+0000 7f56b8a00640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56b40c9500 0x7f56b414eed0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f56a8000c00 tx=0x7f56a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.558+0000 7f56b0c00640 1 -- 172.21.15.28:0/290208438 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a8032440 con 0x7f56b40c9500 2024-09-13T16:45:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.558+0000 7f56b0c00640 1 -- 172.21.15.28:0/290208438 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a8030980 con 0x7f56b40c9500 2024-09-13T16:45:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.558+0000 7f56b0c00640 1 -- 172.21.15.28:0/290208438 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a8030d30 con 0x7f56b40c9500 2024-09-13T16:45:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.558+0000 7f56b9e00640 1 -- 172.21.15.28:0/290208438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b40c8a40 con 0x7f56b40c9500 2024-09-13T16:45:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.558+0000 7f56b9e00640 1 -- 172.21.15.28:0/290208438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b41f2460 con 0x7f56b40c9500 2024-09-13T16:45:34.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.559+0000 7f56b9e00640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b41e4710 0x7f56b40c6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.559+0000 7f56b9e00640 1 -- 172.21.15.28:0/290208438 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b4072440 con 0x7f56b41e4710 2024-09-13T16:45:34.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.560+0000 7f56b3e00640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b41e4710 0x7f56b40c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.560+0000 7f56b3e00640 1 -- 172.21.15.28:0/290208438 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f56b40cea10 con 0x7f56b41e4710 2024-09-13T16:45:34.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.560+0000 7f56b3e00640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b41e4710 0x7f56b40c6030 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f56b4159c60 tx=0x7f56a0011d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:34.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.560+0000 7f56b0c00640 1 -- 172.21.15.28:0/290208438 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a0004300 con 0x7f56b41e4710 2024-09-13T16:45:34.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.561+0000 7f56b0c00640 1 -- 172.21.15.28:0/290208438 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f56a0004460 con 0x7f56b41e4710 2024-09-13T16:45:34.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.561+0000 7f56b0c00640 1 -- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b41e4710 msgr2=0x7f56b40c6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.561+0000 7f56b0c00640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b41e4710 0x7f56b40c6030 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f56b4159c60 tx=0x7f56a0011d40 comp rx=0 tx=0).stop 2024-09-13T16:45:34.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.561+0000 7f56b0c00640 1 -- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b41e4710 msgr2=0x7f56b40c6030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:45:34.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.561+0000 7f56b0c00640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b41e4710 0x7f56b40c6030 secure :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f56b4159c60 tx=0x7f56a0011d40 comp rx=0 tx=0).stop 2024-09-13T16:45:34.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.562+0000 7f56b0c00640 1 -- 172.21.15.28:0/290208438 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f56a8039040 con 0x7f56b40c9500 2024-09-13T16:45:34.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.563+0000 7f56b0c00640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f569406a980 0x7f569406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.563+0000 7f56b0c00640 1 -- 172.21.15.28:0/290208438 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f56a80a9850 con 0x7f56b40c9500 2024-09-13T16:45:34.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.563+0000 7f56b3e00640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f569406a980 0x7f569406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.564+0000 7f56b3e00640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f569406a980 0x7f569406ce40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f56a0018000 tx=0x7f56a0008070 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.574+0000 7f56b9e00640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b40c7ee0 0x7f56b40c6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.574+0000 7f56b9e00640 1 -- 172.21.15.28:0/290208438 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40c6ab0 con 0x7f56b40c7ee0 2024-09-13T16:45:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.574+0000 7f56b9400640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b40c7ee0 0x7f56b40c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.575+0000 7f56b9400640 1 -- 172.21.15.28:0/290208438 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f56b40c9d00 con 0x7f56b40c7ee0 2024-09-13T16:45:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.575+0000 7f56b9400640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b40c7ee0 0x7f56b40c6570 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0012e0 tx=0x7f56ac00ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.575+0000 7f56b0c00640 1 -- 172.21.15.28:0/290208438 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac00ee70 con 0x7f56b40c7ee0 2024-09-13T16:45:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.576+0000 7f56b0c00640 1 -- 172.21.15.28:0/290208438 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4956 (secure 0 0 0) 0x7f56ac003b80 con 0x7f56b40c7ee0 2024-09-13T16:45:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.576+0000 7f56b0c00640 1 -- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b40c7ee0 msgr2=0x7f56b40c6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.576+0000 7f56b0c00640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b40c7ee0 0x7f56b40c6570 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0012e0 tx=0x7f56ac00ec20 comp rx=0 tx=0).stop 2024-09-13T16:45:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.576+0000 7f56b0c00640 1 -- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b40c7ee0 msgr2=0x7f56b40c6570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:45:34.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.576+0000 7f56b0c00640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56b40c7ee0 0x7f56b40c6570 secure :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0012e0 tx=0x7f56ac00ec20 comp rx=0 tx=0).stop 2024-09-13T16:45:34.578 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:45:34.578 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:45:34.578 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:45:34.578 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:45:34.578 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 10, 2024-09-13T16:45:34.578 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:45:34.578 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:45:34.578 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:45:34.578 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:45:34.579 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.579 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 16, 2024-09-13T16:45:34.579 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:45:34.579 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:45:34.579 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:45:34.579 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:34.579 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:34.579 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:34.579 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:34.579 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:34.579 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:34.579 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:34.579 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:34.580 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:34.580 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:34.580 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.580 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:45:34.580 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:45:34.580 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:34.580 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:34.580 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:34.580 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:34.580 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:34.580 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:34.580 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:45:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:45:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:45:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:45:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:45:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:45:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:45:34.581 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:45:34.582 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:45:34.582 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:45:34.582 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:45:34.582 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:45:34.582 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:45:34.582 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:45:34.582 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:45:34.582 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:45:34.582 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:45:34.582 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:34.582 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:34.583 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:34.583 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:34.583 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:34.583 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:34.583 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:34.583 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:34.583 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:34.583 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:34.583 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.583 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:45:34.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.584 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:45:34.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.584 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:45:34.584 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:45:34.584 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:34.584 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:34.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.584 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:34.584 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:45:34.584 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.585 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:34.585 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:45:34.585 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.585 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.585 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.585 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:45:34.586 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:45:34.586 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:34.586 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:34.586 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:34.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.586 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:45:34.586 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:45:34.586 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:34.586 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:34.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.586 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:34.587 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:45:34.587 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.587 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:34.587 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:45:34.587 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.587 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.587 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.588 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:45:34.588 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:45:34.588 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:34.588 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:34.588 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:34.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.588 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:45:34.589 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:45:34.589 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:34.589 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:34.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.589 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:34.589 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:45:34.589 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.589 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:34.590 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:45:34.590 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.590 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.590 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.590 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:45:34.590 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:45:34.590 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:34.591 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:34.591 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:34.591 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.591 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.591 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:45:34.591 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:45:34.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.591 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:34.592 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:34.592 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:45:34.592 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.592 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.592 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:45:34.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.592 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:34.593 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:34.593 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:45:34.593 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.593 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.593 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:45:34.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.593 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:34.594 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:34.594 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4 2024-09-13T16:45:34.594 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.594 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.594 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:45:34.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.594 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:34.595 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:34.595 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:45:34.595 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.595 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.595 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": [ 2024-09-13T16:45:34.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.595 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:34.595 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:34.595 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:45:34.595 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.596 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.596 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:45:34.596 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:45:34.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.576+0000 7f56b9e00640 1 -- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f569406a980 msgr2=0x7f569406ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.576+0000 7f56b9e00640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f569406a980 0x7f569406ce40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f56a0018000 tx=0x7f56a0008070 comp rx=0 tx=0).stop 2024-09-13T16:45:34.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.576+0000 7f56b9e00640 1 -- 172.21.15.28:0/290208438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56b40c9500 msgr2=0x7f56b414eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.576+0000 7f56b9e00640 1 --2- 172.21.15.28:0/290208438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56b40c9500 0x7f56b414eed0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f56a8000c00 tx=0x7f56a80046c0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.576+0000 7f56b9400640 1 -- 172.21.15.28:0/290208438 reap_dead start 2024-09-13T16:45:34.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.576+0000 7f56b9e00640 1 -- 172.21.15.28:0/290208438 shutdown_connections 2024-09-13T16:45:34.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.576+0000 7f56b9e00640 1 -- 172.21.15.28:0/290208438 >> 172.21.15.28:0/290208438 conn(0x7f56b406c180 msgr2=0x7f56b40c3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:34.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.577+0000 7f56b9e00640 1 -- 172.21.15.28:0/290208438 shutdown_connections 2024-09-13T16:45:34.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.577+0000 7f56b9e00640 1 -- 172.21.15.28:0/290208438 wait complete. 2024-09-13T16:45:34.599 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:45:34.599 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-13T16:45:34.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T16:45:34.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.777+0000 7fc9ee000640 1 Processor -- start 2024-09-13T16:45:34.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.777+0000 7fc9ee000640 1 -- start start 2024-09-13T16:45:34.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.778+0000 7fc9ee000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc9e80c2900 0x7fc9e80c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.778+0000 7fc9ee000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e814fcf0 con 0x7fc9e814f910 2024-09-13T16:45:34.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.778+0000 7fc9ee000640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e80c6850 con 0x7fc9e8071cd0 2024-09-13T16:45:34.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.778+0000 7fc9ee000640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9e80c2e00 con 0x7fc9e80c2900 2024-09-13T16:45:34.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.779+0000 7fc9ecc00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc9e814f910 0x7fc9e8071140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48056/0 (socket says 172.21.15.28:48056) 2024-09-13T16:45:34.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.779+0000 7fc9ecc00640 1 -- 172.21.15.28:0/2543966155 learned_addr learned my addr 172.21.15.28:0/2543966155 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:45:34.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.779+0000 7fc9e7e00640 1 --2- 172.21.15.28:0/2543966155 >> v2:172.21.15.191:3300/0 conn(0x7fc9e80c2900 0x7fc9e80c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.779+0000 7fc9e7e00640 1 -- 172.21.15.28:0/2543966155 >> v1:172.21.15.191:6789/0 conn(0x7fc9e8071cd0 legacy=0x7fc9e815ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.779+0000 7fc9e7e00640 1 -- 172.21.15.28:0/2543966155 >> v1:172.21.15.28:6789/0 conn(0x7fc9e814f910 legacy=0x7fc9e8071140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.779+0000 7fc9e7e00640 1 -- 172.21.15.28:0/2543966155 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e80c52b0 con 0x7fc9e80c2900 2024-09-13T16:45:34.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.780+0000 7fc9e7e00640 1 --2- 172.21.15.28:0/2543966155 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9e80c2900 0x7fc9e80c6ce0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc9d40097a0 tx=0x7fc9d402a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e3143b7afe684e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:34.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.780+0000 7fc9e7400640 1 -- 172.21.15.28:0/2543966155 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d4032440 con 0x7fc9e80c2900 2024-09-13T16:45:34.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.781+0000 7fc9e7400640 1 -- 172.21.15.28:0/2543966155 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d40325a0 con 0x7fc9e80c2900 2024-09-13T16:45:34.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.781+0000 7fc9e7400640 1 -- 172.21.15.28:0/2543966155 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d4032930 con 0x7fc9e80c2900 2024-09-13T16:45:34.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.781+0000 7fc9ee000640 1 -- 172.21.15.28:0/2543966155 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9e80c2900 msgr2=0x7fc9e80c6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.781+0000 7fc9ee000640 1 --2- 172.21.15.28:0/2543966155 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9e80c2900 0x7fc9e80c6ce0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc9d40097a0 tx=0x7fc9d402a230 comp rx=0 tx=0).stop 2024-09-13T16:45:34.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.781+0000 7fc9ee000640 1 -- 172.21.15.28:0/2543966155 shutdown_connections 2024-09-13T16:45:34.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.781+0000 7fc9ee000640 1 --2- 172.21.15.28:0/2543966155 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9e80c2900 0x7fc9e80c6ce0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.782+0000 7fc9ee000640 1 -- 172.21.15.28:0/2543966155 >> 172.21.15.28:0/2543966155 conn(0x7fc9e8066360 msgr2=0x7fc9e806ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:34.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.782+0000 7fc9ee000640 1 -- 172.21.15.28:0/2543966155 shutdown_connections 2024-09-13T16:45:34.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.782+0000 7fc9ee000640 1 -- 172.21.15.28:0/2543966155 wait complete. 2024-09-13T16:45:34.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.783+0000 7fc9ee000640 1 Processor -- start 2024-09-13T16:45:34.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.784+0000 7fc9ee000640 1 -- start start 2024-09-13T16:45:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.785+0000 7fc9ee000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e8071cd0 0x7fc9e81f6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.785+0000 7fc9ee000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9e80c2900 0x7fc9e8153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.785+0000 7fc9ee000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9e814f910 0x7fc9e8145080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.785+0000 7fc9ee000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e81537c0 con 0x7fc9e80c2900 2024-09-13T16:45:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.785+0000 7fc9ee000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e806bd60 con 0x7fc9e814f910 2024-09-13T16:45:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.785+0000 7fc9ee000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e8153a30 con 0x7fc9e8071cd0 2024-09-13T16:45:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.785+0000 7fc9ecc00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e8071cd0 0x7fc9e81f6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.785+0000 7fc9e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9e80c2900 0x7fc9e8153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.785+0000 7fc9ed600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9e814f910 0x7fc9e8145080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.785+0000 7fc9ecc00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e8071cd0 0x7fc9e81f6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:45134/0 (socket says 172.21.15.28:45134) 2024-09-13T16:45:34.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.785+0000 7fc9ecc00640 1 -- 172.21.15.28:0/4155850889 learned_addr learned my addr 172.21.15.28:0/4155850889 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:34.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.786+0000 7fc9ed600640 1 -- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e8071cd0 msgr2=0x7fc9e81f6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.786+0000 7fc9ed600640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e8071cd0 0x7fc9e81f6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.786+0000 7fc9ed600640 1 -- 172.21.15.28:0/4155850889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9e80c2900 msgr2=0x7fc9e8153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.786+0000 7fc9ed600640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9e80c2900 0x7fc9e8153ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.786+0000 7fc9ed600640 1 -- 172.21.15.28:0/4155850889 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d4009400 con 0x7fc9e814f910 2024-09-13T16:45:34.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.786+0000 7fc9ecc00640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e8071cd0 0x7fc9e81f6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:45:34.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.786+0000 7fc9e7e00640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9e80c2900 0x7fc9e8153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:45:34.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.786+0000 7fc9ed600640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9e814f910 0x7fc9e8145080 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000c690 tx=0x7fc9e000cb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:34.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.787+0000 7fc9e4c00640 1 -- 172.21.15.28:0/4155850889 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0003b70 con 0x7fc9e814f910 2024-09-13T16:45:34.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.787+0000 7fc9e4c00640 1 -- 172.21.15.28:0/4155850889 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e0003cd0 con 0x7fc9e814f910 2024-09-13T16:45:34.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.787+0000 7fc9e4c00640 1 -- 172.21.15.28:0/4155850889 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0005310 con 0x7fc9e814f910 2024-09-13T16:45:34.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.787+0000 7fc9ee000640 1 -- 172.21.15.28:0/4155850889 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e8150590 con 0x7fc9e814f910 2024-09-13T16:45:34.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.787+0000 7fc9ee000640 1 -- 172.21.15.28:0/4155850889 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e814ffd0 con 0x7fc9e814f910 2024-09-13T16:45:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.788+0000 7fc9ee000640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e813fa10 0x7fc9e813fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.788+0000 7fc9ee000640 1 -- 172.21.15.28:0/4155850889 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e8066ad0 con 0x7fc9e813fa10 2024-09-13T16:45:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.789+0000 7fc9ecc00640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e813fa10 0x7fc9e813fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.789+0000 7fc9ecc00640 1 -- 172.21.15.28:0/4155850889 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9e814a270 con 0x7fc9e813fa10 2024-09-13T16:45:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.789+0000 7fc9e4c00640 1 -- 172.21.15.28:0/4155850889 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc9e0005470 con 0x7fc9e814f910 2024-09-13T16:45:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.790+0000 7fc9ecc00640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e813fa10 0x7fc9e813fdf0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc00fae0 tx=0x7fc9dc001de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.790+0000 7fc9e4c00640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc9c806ac00 0x7fc9c806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.790+0000 7fc9e4c00640 1 -- 172.21.15.28:0/4155850889 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc9e0089000 con 0x7fc9e814f910 2024-09-13T16:45:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.791+0000 7fc9e7e00640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc9c806ac00 0x7fc9c806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.791+0000 7fc9e4c00640 1 -- 172.21.15.28:0/4155850889 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc005ac0 con 0x7fc9e813fa10 2024-09-13T16:45:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.791+0000 7fc9e4c00640 1 -- 172.21.15.28:0/4155850889 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fc9dc005c20 con 0x7fc9e813fa10 2024-09-13T16:45:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.791+0000 7fc9e4c00640 1 -- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e813fa10 msgr2=0x7fc9e813fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.791+0000 7fc9e4c00640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e813fa10 0x7fc9e813fdf0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc00fae0 tx=0x7fc9dc001de0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.791+0000 7fc9e4c00640 1 -- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e813fa10 msgr2=0x7fc9e813fdf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:45:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.791+0000 7fc9e4c00640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e813fa10 0x7fc9e813fdf0 secure :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc00fae0 tx=0x7fc9dc001de0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.791+0000 7fc9e7e00640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc9c806ac00 0x7fc9c806d0c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4034000 tx=0x7fc9d402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.804+0000 7fc9ee000640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e8140330 0x7fc9e8069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.804+0000 7fc9ee000640 1 -- 172.21.15.28:0/4155850889 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e80c5830 con 0x7fc9e8140330 2024-09-13T16:45:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.805+0000 7fc9ecc00640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e8140330 0x7fc9e8069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.806+0000 7fc9ecc00640 1 -- 172.21.15.28:0/4155850889 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc9e80c64e0 con 0x7fc9e8140330 2024-09-13T16:45:34.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.807+0000 7fc9ecc00640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e8140330 0x7fc9e8069e80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc00fae0 tx=0x7fc9dc000fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.808+0000 7fc9e4c00640 1 -- 172.21.15.28:0/4155850889 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc023070 con 0x7fc9e8140330 2024-09-13T16:45:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.808+0000 7fc9e4c00640 1 -- 172.21.15.28:0/4155850889 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4458 (secure 0 0 0) 0x7fc9dc0073f0 con 0x7fc9e8140330 2024-09-13T16:45:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.808+0000 7fc9e4c00640 1 -- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e8140330 msgr2=0x7fc9e8069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.808+0000 7fc9e4c00640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e8140330 0x7fc9e8069e80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc00fae0 tx=0x7fc9dc000fb0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.808+0000 7fc9e4c00640 1 -- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e8140330 msgr2=0x7fc9e8069e80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:45:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.808+0000 7fc9e4c00640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9e8140330 0x7fc9e8069e80 secure :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc00fae0 tx=0x7fc9dc000fb0 comp rx=0 tx=0).stop 2024-09-13T16:45:34.811 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:45:34.811 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:45:34.811 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:45:34.811 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:45:34.811 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 10, 2024-09-13T16:45:34.811 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:45:34.811 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:45:34.811 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:45:34.811 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:45:34.811 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.811 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 16, 2024-09-13T16:45:34.812 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:45:34.812 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:45:34.812 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:45:34.812 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:34.812 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:34.812 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:34.812 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:34.812 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:34.812 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:34.812 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:34.812 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:34.812 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:34.812 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:34.813 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.813 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:45:34.813 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:45:34.813 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:34.813 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:34.813 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:34.813 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:34.813 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:34.813 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:34.813 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:34.814 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:34.814 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:34.814 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:34.814 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.814 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:45:34.814 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:45:34.814 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:45:34.815 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:45:34.815 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:45:34.815 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:45:34.815 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:45:34.815 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:45:34.815 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:45:34.815 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:45:34.815 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:45:34.815 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:45:34.815 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:45:34.815 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:45:34.815 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:45:34.815 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:45:34.815 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:45:34.816 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:34.816 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:34.816 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:34.816 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:34.816 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:34.816 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:34.816 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:34.816 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:34.816 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:34.816 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:34.817 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.817 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:45:34.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.817 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:45:34.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.817 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:45:34.817 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:45:34.817 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:34.817 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:34.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:34.817 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:45:34.818 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:34.818 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:45:34.818 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.818 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.818 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.818 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:45:34.818 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:45:34.818 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:34.818 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:34.818 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:34.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.819 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:45:34.819 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:45:34.819 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:34.819 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:34.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:34.819 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:45:34.819 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:34.819 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:45:34.820 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.820 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.820 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.820 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:45:34.820 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:45:34.820 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:34.820 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:34.820 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:34.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.820 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:45:34.820 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:45:34.821 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:34.821 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:34.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:34.821 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:45:34.821 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:34.821 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:45:34.821 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:34.821 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.821 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.822 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:45:34.822 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:45:34.822 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:34.822 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:34.822 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:34.822 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.822 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.822 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:45:34.822 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:45:34.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.822 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:34.822 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:34.822 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:45:34.823 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.823 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:34.823 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:45:34.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:34.823 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:34.823 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:34.823 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:45:34.823 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:34.823 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:34.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:34.823 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:45:34.823 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:45:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.809+0000 7fc9ee000640 1 -- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc9c806ac00 msgr2=0x7fc9c806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.809+0000 7fc9ee000640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc9c806ac00 0x7fc9c806d0c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4034000 tx=0x7fc9d402e000 comp rx=0 tx=0).stop 2024-09-13T16:45:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.809+0000 7fc9ee000640 1 -- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9e814f910 msgr2=0x7fc9e8145080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.809+0000 7fc9ee000640 1 --2- 172.21.15.28:0/4155850889 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9e814f910 0x7fc9e8145080 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000c690 tx=0x7fc9e000cb50 comp rx=0 tx=0).stop 2024-09-13T16:45:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.809+0000 7fc9ed600640 1 -- 172.21.15.28:0/4155850889 reap_dead start 2024-09-13T16:45:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.809+0000 7fc9ed600640 1 -- 172.21.15.28:0/4155850889 reap_dead start 2024-09-13T16:45:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.809+0000 7fc9ee000640 1 -- 172.21.15.28:0/4155850889 shutdown_connections 2024-09-13T16:45:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.809+0000 7fc9ee000640 1 -- 172.21.15.28:0/4155850889 >> 172.21.15.28:0/4155850889 conn(0x7fc9e8066360 msgr2=0x7fc9e814bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.809+0000 7fc9ee000640 1 -- 172.21.15.28:0/4155850889 shutdown_connections 2024-09-13T16:45:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:34.809+0000 7fc9ee000640 1 -- 172.21.15.28:0/4155850889 wait complete. 2024-09-13T16:45:34.826 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:45:34.826 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-13T16:45:34.827 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-13T16:45:34.827 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-13T16:45:34.827 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-13T16:45:34.827 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-13T16:45:34.827 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:45:34.845 INFO:tasks.ceph.mon.b:Stopped 2024-09-13T16:45:34.845 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-13T16:45:34.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:45:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.004+0000 7f01b0c00640 1 Processor -- start 2024-09-13T16:45:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.004+0000 7f01b0c00640 1 -- start start 2024-09-13T16:45:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.005+0000 7f01b0c00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f01ac15d280 0x7f01ac0713b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.006+0000 7f01b0c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ac15e030 con 0x7f01ac15cea0 2024-09-13T16:45:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.006+0000 7f01b0c00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ac06d950 con 0x7f01ac15f570 2024-09-13T16:45:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.006+0000 7f01b0c00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ac0718f0 con 0x7f01ac15d280 2024-09-13T16:45:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.006+0000 7f01abe00640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f01ac15f570 legacy=0x7f01ac15f980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-09-13T16:45:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.006+0000 7f01aaa00640 1 -- >> v2:172.21.15.191:3300/0 conn(0x7f01ac15d280 msgr2=0x7f01ac0713b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3300/0 2024-09-13T16:45:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.006+0000 7f01aaa00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f01ac15d280 0x7f01ac0713b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.006+0000 7f01ab400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f01ac15cea0 0x7f01ac06d030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48072/0 (socket says 172.21.15.28:48072) 2024-09-13T16:45:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.006+0000 7f01ab400640 1 -- 172.21.15.28:0/1656296001 learned_addr learned my addr 172.21.15.28:0/1656296001 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:45:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.007+0000 7f01aa000640 1 -- 172.21.15.28:0/1656296001 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186176347 0 0) 0x7f01ac15e030 con 0x7f01ac15cea0 2024-09-13T16:45:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.007+0000 7f01aa000640 1 -- 172.21.15.28:0/1656296001 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0190003520 con 0x7f01ac15cea0 2024-09-13T16:45:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.007+0000 7f01aa000640 1 -- 172.21.15.28:0/1656296001 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3793455585 0 0) 0x7f0190003520 con 0x7f01ac15cea0 2024-09-13T16:45:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.007+0000 7f01aa000640 1 -- 172.21.15.28:0/1656296001 >> v2:172.21.15.191:3300/0 conn(0x7f01ac15d280 msgr2=0x7f01ac0713b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:45:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.007+0000 7f01aa000640 1 --2- 172.21.15.28:0/1656296001 >> v2:172.21.15.191:3300/0 conn(0x7f01ac15d280 0x7f01ac0713b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.007+0000 7f01aa000640 1 -- 172.21.15.28:0/1656296001 >> v1:172.21.15.191:6789/0 conn(0x7f01ac15f570 legacy=0x7f01ac15f980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:45:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.007+0000 7f01aa000640 1 -- 172.21.15.28:0/1656296001 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ac06dc30 con 0x7f01ac15cea0 2024-09-13T16:45:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.007+0000 7f01aa000640 1 -- 172.21.15.28:0/1656296001 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1654382301 0 0) 0x7f019c001670 con 0x7f01ac15cea0 2024-09-13T16:45:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.007+0000 7f01aa000640 1 -- 172.21.15.28:0/1656296001 >> v1:172.21.15.28:6789/0 conn(0x7f01ac15cea0 legacy=0x7f01ac06d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.007+0000 7f01aa000640 1 --2- 172.21.15.28:0/1656296001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0190003970 0x7f0190003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.007+0000 7f01aa000640 1 --2- 172.21.15.28:0/1656296001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01900047c0 0x7f0190004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.007+0000 7f01aa000640 1 --2- 172.21.15.28:0/1656296001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01900051f0 0x7f019000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.007+0000 7f01aa000640 1 -- 172.21.15.28:0/1656296001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019000c110 con 0x7f01900051f0 2024-09-13T16:45:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.007+0000 7f01aa000640 1 -- 172.21.15.28:0/1656296001 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019000c280 con 0x7f01900047c0 2024-09-13T16:45:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.007+0000 7f01aa000640 1 -- 172.21.15.28:0/1656296001 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019000c440 con 0x7f0190003970 2024-09-13T16:45:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.007+0000 7f01aaa00640 1 -- 172.21.15.28:0/1656296001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01900047c0 msgr2=0x7f0190004c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3300/0 2024-09-13T16:45:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.007+0000 7f01aaa00640 1 --2- 172.21.15.28:0/1656296001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01900047c0 0x7f0190004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.008+0000 7f01ab400640 1 --2- 172.21.15.28:0/1656296001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0190003970 0x7f0190003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.008+0000 7f01abe00640 1 --2- 172.21.15.28:0/1656296001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01900051f0 0x7f019000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.008+0000 7f01ab400640 1 -- 172.21.15.28:0/1656296001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01900047c0 msgr2=0x7f0190004c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:45:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.008+0000 7f01ab400640 1 --2- 172.21.15.28:0/1656296001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01900047c0 0x7f0190004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:35.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.008+0000 7f01ab400640 1 -- 172.21.15.28:0/1656296001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01900051f0 msgr2=0x7f019000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:45:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.008+0000 7f01ab400640 1 --2- 172.21.15.28:0/1656296001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01900051f0 0x7f019000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.008+0000 7f01ab400640 1 -- 172.21.15.28:0/1656296001 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0190003520 con 0x7f0190003970 2024-09-13T16:45:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.008+0000 7f01abe00640 1 --2- 172.21.15.28:0/1656296001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01900051f0 0x7f019000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:45:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.008+0000 7f01abe00640 1 -- 172.21.15.28:0/1656296001 reap_dead start 2024-09-13T16:45:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.010+0000 7f01ab400640 1 --2- 172.21.15.28:0/1656296001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0190003970 0x7f0190003d70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f019c00c190 tx=0x7f01900057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e805041aa2907f4a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.010+0000 7f01aa000640 1 -- 172.21.15.28:0/1656296001 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c005ad0 con 0x7f0190003970 2024-09-13T16:45:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.010+0000 7f01aa000640 1 -- 172.21.15.28:0/1656296001 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c02d070 con 0x7f0190003970 2024-09-13T16:45:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.010+0000 7f01aa000640 1 -- 172.21.15.28:0/1656296001 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c005e90 con 0x7f0190003970 2024-09-13T16:45:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.010+0000 7f01b0c00640 1 -- 172.21.15.28:0/1656296001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0190003970 msgr2=0x7f0190003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.010+0000 7f01b0c00640 1 --2- 172.21.15.28:0/1656296001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0190003970 0x7f0190003d70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f019c00c190 tx=0x7f01900057f0 comp rx=0 tx=0).stop 2024-09-13T16:45:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.011+0000 7f01b0c00640 1 -- 172.21.15.28:0/1656296001 shutdown_connections 2024-09-13T16:45:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.011+0000 7f01b0c00640 1 --2- 172.21.15.28:0/1656296001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0190003970 0x7f0190003d70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.011+0000 7f01b0c00640 1 -- 172.21.15.28:0/1656296001 >> 172.21.15.28:0/1656296001 conn(0x7f01ac069e80 msgr2=0x7f01ac06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.011+0000 7f01b0c00640 1 -- 172.21.15.28:0/1656296001 shutdown_connections 2024-09-13T16:45:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.011+0000 7f01b0c00640 1 -- 172.21.15.28:0/1656296001 wait complete. 2024-09-13T16:45:35.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.013+0000 7f01b0c00640 1 Processor -- start 2024-09-13T16:45:35.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.014+0000 7f01b0c00640 1 -- start start 2024-09-13T16:45:35.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.015+0000 7f01b0c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0190003970 0x7f01ac0c0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:35.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.015+0000 7f01b0c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01ac15dd20 0x7f01ac0b94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:35.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.015+0000 7f01b0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac0b99f0 0x7f01ac1f53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:35.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.015+0000 7f01b0c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac0babb0 con 0x7f01ac0b99f0 2024-09-13T16:45:35.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.015+0000 7f01b0c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac0ba7f0 con 0x7f0190003970 2024-09-13T16:45:35.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.015+0000 7f01b0c00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac0c0740 con 0x7f01ac15dd20 2024-09-13T16:45:35.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.015+0000 7f01ab400640 1 -- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0190003970 msgr2=0x7f01ac0c0200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.191:3300/0 2024-09-13T16:45:35.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.015+0000 7f01ab400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0190003970 0x7f01ac0c0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:35.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.016+0000 7f01aaa00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01ac15dd20 0x7f01ac0b94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:35.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.016+0000 7f01aaa00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01ac15dd20 0x7f01ac0b94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:45182/0 (socket says 172.21.15.28:45182) 2024-09-13T16:45:35.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.016+0000 7f01aaa00640 1 -- 172.21.15.28:0/4144421950 learned_addr learned my addr 172.21.15.28:0/4144421950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:35.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.016+0000 7f01abe00640 1 --2- 172.21.15.28:0/4144421950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac0b99f0 0x7f01ac1f53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.016+0000 7f01aaa00640 1 -- 172.21.15.28:0/4144421950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0190003970 msgr2=0x7f01ac0c0200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:45:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.016+0000 7f01aaa00640 1 --2- 172.21.15.28:0/4144421950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0190003970 0x7f01ac0c0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.016+0000 7f01aaa00640 1 -- 172.21.15.28:0/4144421950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac0b99f0 msgr2=0x7f01ac1f53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.017+0000 7f01aaa00640 1 --2- 172.21.15.28:0/4144421950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac0b99f0 0x7f01ac1f53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.017+0000 7f01aaa00640 1 -- 172.21.15.28:0/4144421950 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ac0c0e30 con 0x7f01ac15dd20 2024-09-13T16:45:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.017+0000 7f01abe00640 1 --2- 172.21.15.28:0/4144421950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac0b99f0 0x7f01ac1f53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:45:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.017+0000 7f01aaa00640 1 --2- 172.21.15.28:0/4144421950 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01ac15dd20 0x7f01ac0b94b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f019800b4e0 tx=0x7f019800b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.017+0000 7f01a3400640 1 -- 172.21.15.28:0/4144421950 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0198003d70 con 0x7f01ac15dd20 2024-09-13T16:45:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.017+0000 7f01a3400640 1 -- 172.21.15.28:0/4144421950 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01980050f0 con 0x7f01ac15dd20 2024-09-13T16:45:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.017+0000 7f01a3400640 1 -- 172.21.15.28:0/4144421950 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01980054a0 con 0x7f01ac15dd20 2024-09-13T16:45:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.018+0000 7f01b0c00640 1 -- 172.21.15.28:0/4144421950 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ac0b78c0 con 0x7f01ac15dd20 2024-09-13T16:45:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.019+0000 7f01b0c00640 1 -- 172.21.15.28:0/4144421950 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ac0c10d0 con 0x7f01ac15dd20 2024-09-13T16:45:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.020+0000 7f01b0c00640 1 -- 172.21.15.28:0/4144421950 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ac071330 con 0x7f01ac15dd20 2024-09-13T16:45:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.021+0000 7f01a3400640 1 -- 172.21.15.28:0/4144421950 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0198005600 con 0x7f01ac15dd20 2024-09-13T16:45:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.022+0000 7f01a3400640 1 --2- 172.21.15.28:0/4144421950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f018c06abe0 0x7f018c06d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.022+0000 7f01a3400640 1 -- 172.21.15.28:0/4144421950 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0198088650 con 0x7f01ac15dd20 2024-09-13T16:45:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.022+0000 7f01ab400640 1 --2- 172.21.15.28:0/4144421950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f018c06abe0 0x7f018c06d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:35.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.023+0000 7f01ab400640 1 --2- 172.21.15.28:0/4144421950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f018c06abe0 0x7f018c06d0a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f019c032000 tx=0x7f019c009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.027+0000 7f01a3400640 1 -- 172.21.15.28:0/4144421950 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f01980537b0 con 0x7f01ac15dd20 2024-09-13T16:45:35.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.272+0000 7f01b0c00640 1 -- 172.21.15.28:0/4144421950 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f01ac071740 con 0x7f01ac15dd20 2024-09-13T16:45:35.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.273+0000 7f01a3400640 1 -- 172.21.15.28:0/4144421950 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f0198016620 con 0x7f01ac15dd20 2024-09-13T16:45:35.275 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:45:35.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.275+0000 7f01b0c00640 1 -- 172.21.15.28:0/4144421950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f018c06abe0 msgr2=0x7f018c06d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:35.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.275+0000 7f01b0c00640 1 --2- 172.21.15.28:0/4144421950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f018c06abe0 0x7f018c06d0a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f019c032000 tx=0x7f019c009030 comp rx=0 tx=0).stop 2024-09-13T16:45:35.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.275+0000 7f01b0c00640 1 -- 172.21.15.28:0/4144421950 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01ac15dd20 msgr2=0x7f01ac0b94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:35.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.275+0000 7f01b0c00640 1 --2- 172.21.15.28:0/4144421950 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01ac15dd20 0x7f01ac0b94b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f019800b4e0 tx=0x7f019800b8e0 comp rx=0 tx=0).stop 2024-09-13T16:45:35.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.276+0000 7f01b0c00640 1 -- 172.21.15.28:0/4144421950 shutdown_connections 2024-09-13T16:45:35.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.276+0000 7f01b0c00640 1 --2- 172.21.15.28:0/4144421950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f018c06abe0 0x7f018c06d0a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:35.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.276+0000 7f01b0c00640 1 --2- 172.21.15.28:0/4144421950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac0b99f0 0x7f01ac1f53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:35.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.276+0000 7f01b0c00640 1 --2- 172.21.15.28:0/4144421950 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01ac15dd20 0x7f01ac0b94b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:35.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.276+0000 7f01b0c00640 1 --2- 172.21.15.28:0/4144421950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0190003970 0x7f01ac0c0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:35.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.276+0000 7f01b0c00640 1 -- 172.21.15.28:0/4144421950 >> 172.21.15.28:0/4144421950 conn(0x7f01ac069e80 msgr2=0x7f01ac0c2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:35.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.276+0000 7f01b0c00640 1 -- 172.21.15.28:0/4144421950 shutdown_connections 2024-09-13T16:45:35.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:35.276+0000 7f01b0c00640 1 -- 172.21.15.28:0/4144421950 wait complete. 2024-09-13T16:45:38.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:45:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.451+0000 7f4667e00640 1 Processor -- start 2024-09-13T16:45:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.452+0000 7f4667e00640 1 -- start start 2024-09-13T16:45:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.453+0000 7f4667e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46680cadb0 0x7f46680cb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.453+0000 7f4667e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466815c040 con 0x7f4668152d70 2024-09-13T16:45:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.453+0000 7f4667e00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466815c2e0 con 0x7f46680ce650 2024-09-13T16:45:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.453+0000 7f4667e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466815c5a0 con 0x7f46680cadb0 2024-09-13T16:45:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.453+0000 7f4666000640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f46680ce650 legacy=0x7f46680cea30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-09-13T16:45:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.453+0000 7f4666a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4668152d70 0x7f46680c8d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48078/0 (socket says 172.21.15.28:48078) 2024-09-13T16:45:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.453+0000 7f4666a00640 1 -- 172.21.15.28:0/1044253049 learned_addr learned my addr 172.21.15.28:0/1044253049 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:45:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.453+0000 7f4667400640 1 --2- 172.21.15.28:0/1044253049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46680cadb0 0x7f46680cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:38.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.454+0000 7f4665600640 1 -- 172.21.15.28:0/1044253049 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412229963 0 0) 0x7f466815c040 con 0x7f4668152d70 2024-09-13T16:45:38.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.454+0000 7f4665600640 1 -- 172.21.15.28:0/1044253049 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f464c003520 con 0x7f4668152d70 2024-09-13T16:45:38.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.454+0000 7f4665600640 1 -- 172.21.15.28:0/1044253049 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4238603830 0 0) 0x7f464c003520 con 0x7f4668152d70 2024-09-13T16:45:38.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.454+0000 7f4665600640 1 -- 172.21.15.28:0/1044253049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46680cadb0 msgr2=0x7f46680cb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:45:38.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.454+0000 7f4665600640 1 --2- 172.21.15.28:0/1044253049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46680cadb0 0x7f46680cb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:38.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.454+0000 7f4665600640 1 -- 172.21.15.28:0/1044253049 >> v1:172.21.15.191:6789/0 conn(0x7f46680ce650 legacy=0x7f46680cea30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:45:38.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.454+0000 7f4665600640 1 -- 172.21.15.28:0/1044253049 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466815c880 con 0x7f4668152d70 2024-09-13T16:45:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.454+0000 7f4665600640 1 -- 172.21.15.28:0/1044253049 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1654382301 0 0) 0x7f4658001670 con 0x7f4668152d70 2024-09-13T16:45:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.455+0000 7f4665600640 1 -- 172.21.15.28:0/1044253049 >> v1:172.21.15.28:6789/0 conn(0x7f4668152d70 legacy=0x7f46680c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.455+0000 7f4665600640 1 --2- 172.21.15.28:0/1044253049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f464c0039a0 0x7f464c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.455+0000 7f4665600640 1 --2- 172.21.15.28:0/1044253049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46680cadb0 0x7f464c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.455+0000 7f4665600640 1 --2- 172.21.15.28:0/1044253049 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f464c004d00 0x7f464c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.455+0000 7f4665600640 1 -- 172.21.15.28:0/1044253049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466815c5a0 con 0x7f464c0039a0 2024-09-13T16:45:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.455+0000 7f4665600640 1 -- 172.21.15.28:0/1044253049 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c00ba30 con 0x7f464c004d00 2024-09-13T16:45:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.455+0000 7f4665600640 1 -- 172.21.15.28:0/1044253049 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464c00bbf0 con 0x7f46680cadb0 2024-09-13T16:45:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.455+0000 7f4666000640 1 --2- 172.21.15.28:0/1044253049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46680cadb0 0x7f464c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.455+0000 7f4667400640 1 -- 172.21.15.28:0/1044253049 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f464c004d00 msgr2=0x7f464c005180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3300/0 2024-09-13T16:45:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.455+0000 7f4667400640 1 --2- 172.21.15.28:0/1044253049 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f464c004d00 0x7f464c005180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.455+0000 7f4666a00640 1 --2- 172.21.15.28:0/1044253049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f464c0039a0 0x7f464c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.455+0000 7f4666000640 1 -- 172.21.15.28:0/1044253049 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f464c004d00 msgr2=0x7f464c005180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:45:38.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.455+0000 7f4666000640 1 --2- 172.21.15.28:0/1044253049 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f464c004d00 0x7f464c005180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:38.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.455+0000 7f4666000640 1 -- 172.21.15.28:0/1044253049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f464c0039a0 msgr2=0x7f464c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:45:38.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.455+0000 7f4666000640 1 --2- 172.21.15.28:0/1044253049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f464c0039a0 0x7f464c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:38.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.456+0000 7f4666000640 1 -- 172.21.15.28:0/1044253049 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466815c880 con 0x7f46680cadb0 2024-09-13T16:45:38.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.457+0000 7f4666000640 1 --2- 172.21.15.28:0/1044253049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46680cadb0 0x7f464c0047c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4654009900 tx=0x7f464c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98e70f9a6fc8fa8b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:38.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.457+0000 7f4665600640 1 -- 172.21.15.28:0/1044253049 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4654003bb0 con 0x7f46680cadb0 2024-09-13T16:45:38.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.457+0000 7f4665600640 1 -- 172.21.15.28:0/1044253049 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4654003d10 con 0x7f46680cadb0 2024-09-13T16:45:38.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.457+0000 7f4665600640 1 -- 172.21.15.28:0/1044253049 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465402abe0 con 0x7f46680cadb0 2024-09-13T16:45:38.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.458+0000 7f4667e00640 1 -- 172.21.15.28:0/1044253049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46680cadb0 msgr2=0x7f464c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:38.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.458+0000 7f4667e00640 1 --2- 172.21.15.28:0/1044253049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46680cadb0 0x7f464c0047c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4654009900 tx=0x7f464c0057f0 comp rx=0 tx=0).stop 2024-09-13T16:45:38.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.458+0000 7f4667400640 1 -- 172.21.15.28:0/1044253049 reap_dead start 2024-09-13T16:45:38.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.458+0000 7f4667e00640 1 -- 172.21.15.28:0/1044253049 shutdown_connections 2024-09-13T16:45:38.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.458+0000 7f4667e00640 1 -- 172.21.15.28:0/1044253049 >> 172.21.15.28:0/1044253049 conn(0x7f466806bbf0 msgr2=0x7f46680715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:38.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.458+0000 7f4667e00640 1 -- 172.21.15.28:0/1044253049 shutdown_connections 2024-09-13T16:45:38.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.458+0000 7f4667e00640 1 -- 172.21.15.28:0/1044253049 wait complete. 2024-09-13T16:45:38.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.461+0000 7f4667e00640 1 Processor -- start 2024-09-13T16:45:38.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.461+0000 7f4667e00640 1 -- start start 2024-09-13T16:45:38.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.462+0000 7f4667e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46680cadb0 0x7f466814d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:38.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.462+0000 7f4667e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46680c57e0 0x7f46681f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.462+0000 7f4667e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46680c7f10 0x7f4668154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.462+0000 7f4667e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466814ea80 con 0x7f46680c7f10 2024-09-13T16:45:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.462+0000 7f4667e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466814e270 con 0x7f46680cadb0 2024-09-13T16:45:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.463+0000 7f4667e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46681f2af0 con 0x7f46680c57e0 2024-09-13T16:45:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.463+0000 7f4666a00640 1 -- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46680cadb0 msgr2=0x7f466814d920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.191:3300/0 2024-09-13T16:45:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.463+0000 7f4666a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46680cadb0 0x7f466814d920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.463+0000 7f4667400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46680c7f10 0x7f4668154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.463+0000 7f4666000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46680c57e0 0x7f46681f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.463+0000 7f4666000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46680c57e0 0x7f46681f25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:45208/0 (socket says 172.21.15.28:45208) 2024-09-13T16:45:38.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.463+0000 7f4666000640 1 -- 172.21.15.28:0/3741652932 learned_addr learned my addr 172.21.15.28:0/3741652932 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.463+0000 7f4667400640 1 -- 172.21.15.28:0/3741652932 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46680c57e0 msgr2=0x7f46681f25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.463+0000 7f4667400640 1 --2- 172.21.15.28:0/3741652932 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46680c57e0 0x7f46681f25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.463+0000 7f4667400640 1 -- 172.21.15.28:0/3741652932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46680cadb0 msgr2=0x7f466814d920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:45:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.463+0000 7f4667400640 1 --2- 172.21.15.28:0/3741652932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46680cadb0 0x7f466814d920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.463+0000 7f4667400640 1 -- 172.21.15.28:0/3741652932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466814e4c0 con 0x7f46680c7f10 2024-09-13T16:45:38.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.464+0000 7f4667400640 1 --2- 172.21.15.28:0/3741652932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46680c7f10 0x7f4668154aa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f46680cf420 tx=0x7f466000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:38.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.464+0000 7f465ea00640 1 -- 172.21.15.28:0/3741652932 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4660015400 con 0x7f46680c7f10 2024-09-13T16:45:38.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.464+0000 7f465ea00640 1 -- 172.21.15.28:0/3741652932 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466000f040 con 0x7f46680c7f10 2024-09-13T16:45:38.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.464+0000 7f465ea00640 1 -- 172.21.15.28:0/3741652932 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46600157e0 con 0x7f46680c7f10 2024-09-13T16:45:38.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.464+0000 7f4667e00640 1 -- 172.21.15.28:0/3741652932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466814e7a0 con 0x7f46680c7f10 2024-09-13T16:45:38.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.464+0000 7f4667e00640 1 -- 172.21.15.28:0/3741652932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46681f2090 con 0x7f46680c7f10 2024-09-13T16:45:38.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.466+0000 7f4667e00640 1 -- 172.21.15.28:0/3741652932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46681f1c10 con 0x7f46680c7f10 2024-09-13T16:45:38.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.468+0000 7f465ea00640 1 -- 172.21.15.28:0/3741652932 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4660015940 con 0x7f46680c7f10 2024-09-13T16:45:38.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.468+0000 7f465ea00640 1 --2- 172.21.15.28:0/3741652932 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f464806abe0 0x7f464806d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:38.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.469+0000 7f465ea00640 1 -- 172.21.15.28:0/3741652932 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4660089360 con 0x7f46680c7f10 2024-09-13T16:45:38.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.469+0000 7f4666a00640 1 --2- 172.21.15.28:0/3741652932 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f464806abe0 0x7f464806d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:38.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.470+0000 7f4666a00640 1 --2- 172.21.15.28:0/3741652932 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f464806abe0 0x7f464806d0a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4658000f60 tx=0x7f4658000ed0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:38.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.477+0000 7f465ea00640 1 -- 172.21.15.28:0/3741652932 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f46681f1c10 con 0x7f46680c7f10 2024-09-13T16:45:38.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.726+0000 7f4667e00640 1 -- 172.21.15.28:0/3741652932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f46681f1c10 con 0x7f46680c7f10 2024-09-13T16:45:38.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.726+0000 7f465ea00640 1 -- 172.21.15.28:0/3741652932 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f46681f1c10 con 0x7f46680c7f10 2024-09-13T16:45:38.728 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:45:38.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.730+0000 7f4667e00640 1 -- 172.21.15.28:0/3741652932 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f464806abe0 msgr2=0x7f464806d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:38.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.730+0000 7f4667e00640 1 --2- 172.21.15.28:0/3741652932 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f464806abe0 0x7f464806d0a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4658000f60 tx=0x7f4658000ed0 comp rx=0 tx=0).stop 2024-09-13T16:45:38.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.730+0000 7f4667e00640 1 -- 172.21.15.28:0/3741652932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46680c7f10 msgr2=0x7f4668154aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:38.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.730+0000 7f4667e00640 1 --2- 172.21.15.28:0/3741652932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46680c7f10 0x7f4668154aa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f46680cf420 tx=0x7f466000d560 comp rx=0 tx=0).stop 2024-09-13T16:45:38.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.730+0000 7f4667e00640 1 -- 172.21.15.28:0/3741652932 shutdown_connections 2024-09-13T16:45:38.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.730+0000 7f4667e00640 1 --2- 172.21.15.28:0/3741652932 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f464806abe0 0x7f464806d0a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:38.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.730+0000 7f4667e00640 1 --2- 172.21.15.28:0/3741652932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46680c7f10 0x7f4668154aa0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:38.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.730+0000 7f4667e00640 1 --2- 172.21.15.28:0/3741652932 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46680c57e0 0x7f46681f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:38.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.730+0000 7f4667e00640 1 --2- 172.21.15.28:0/3741652932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46680cadb0 0x7f466814d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:38.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.730+0000 7f4667e00640 1 -- 172.21.15.28:0/3741652932 >> 172.21.15.28:0/3741652932 conn(0x7f466806bbf0 msgr2=0x7f46680b3100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:38.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.730+0000 7f4667e00640 1 -- 172.21.15.28:0/3741652932 shutdown_connections 2024-09-13T16:45:38.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:38.730+0000 7f4667e00640 1 -- 172.21.15.28:0/3741652932 wait complete. 2024-09-13T16:45:41.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:45:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.924+0000 7f8014a00640 1 Processor -- start 2024-09-13T16:45:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.925+0000 7f8014a00640 1 -- start start 2024-09-13T16:45:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.926+0000 7f8014a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f80100c5440 0x7f80100c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.926+0000 7f8014a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8010156610 0x7f8010150400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.926+0000 7f8014a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80100c5df0 con 0x7f80100c5060 2024-09-13T16:45:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.926+0000 7f8014a00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80100c2ab0 con 0x7f80100c5440 2024-09-13T16:45:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.926+0000 7f8014a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80100c2d70 con 0x7f8010156610 2024-09-13T16:45:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.926+0000 7f800ea00640 1 -- >> v2:172.21.15.191:3300/0 conn(0x7f80100c5440 msgr2=0x7f80100c44d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3300/0 2024-09-13T16:45:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.926+0000 7f800ea00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f80100c5440 0x7f80100c44d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.926+0000 7f800f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f80100c5060 0x7f80100c2300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44856/0 (socket says 172.21.15.28:44856) 2024-09-13T16:45:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.926+0000 7f800f400640 1 -- 172.21.15.28:0/464157776 learned_addr learned my addr 172.21.15.28:0/464157776 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:45:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.927+0000 7f800fe00640 1 --2- 172.21.15.28:0/464157776 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8010156610 0x7f8010150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.927+0000 7f800fe00640 1 -- 172.21.15.28:0/464157776 >> v2:172.21.15.191:3300/0 conn(0x7f80100c5440 msgr2=0x7f80100c44d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:45:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.927+0000 7f800fe00640 1 --2- 172.21.15.28:0/464157776 >> v2:172.21.15.191:3300/0 conn(0x7f80100c5440 0x7f80100c44d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.927+0000 7f800fe00640 1 -- 172.21.15.28:0/464157776 >> v1:172.21.15.28:6789/0 conn(0x7f80100c5060 legacy=0x7f80100c2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.927+0000 7f800fe00640 1 -- 172.21.15.28:0/464157776 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80100c6040 con 0x7f8010156610 2024-09-13T16:45:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.929+0000 7f800fe00640 1 --2- 172.21.15.28:0/464157776 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8010156610 0x7f8010150400 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8008009d90 tx=0x7f800802a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=769238ab9945e5f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.929+0000 7f800e000640 1 -- 172.21.15.28:0/464157776 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8008007740 con 0x7f8010156610 2024-09-13T16:45:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.929+0000 7f800e000640 1 -- 172.21.15.28:0/464157776 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800802d050 con 0x7f8010156610 2024-09-13T16:45:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.929+0000 7f800e000640 1 -- 172.21.15.28:0/464157776 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8008007ac0 con 0x7f8010156610 2024-09-13T16:45:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.930+0000 7f8014a00640 1 -- 172.21.15.28:0/464157776 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8010156610 msgr2=0x7f8010150400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.930+0000 7f8014a00640 1 --2- 172.21.15.28:0/464157776 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8010156610 0x7f8010150400 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8008009d90 tx=0x7f800802a710 comp rx=0 tx=0).stop 2024-09-13T16:45:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.930+0000 7f8014a00640 1 -- 172.21.15.28:0/464157776 shutdown_connections 2024-09-13T16:45:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.930+0000 7f8014a00640 1 --2- 172.21.15.28:0/464157776 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8010156610 0x7f8010150400 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:41.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.930+0000 7f8014a00640 1 --2- 172.21.15.28:0/464157776 >> v2:172.21.15.191:3300/0 conn(0x7f80100c5440 0x7f80100c44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:41.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.930+0000 7f8014a00640 1 -- 172.21.15.28:0/464157776 >> 172.21.15.28:0/464157776 conn(0x7f8010069e80 msgr2=0x7f801006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:41.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.930+0000 7f8014a00640 1 -- 172.21.15.28:0/464157776 shutdown_connections 2024-09-13T16:45:41.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.930+0000 7f8014a00640 1 -- 172.21.15.28:0/464157776 wait complete. 2024-09-13T16:45:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.932+0000 7f8014a00640 1 Processor -- start 2024-09-13T16:45:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.932+0000 7f8014a00640 1 -- start start 2024-09-13T16:45:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.933+0000 7f8014a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100c5060 0x7f80101ef590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.933+0000 7f8014a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80100c5440 0x7f80101ee960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.933+0000 7f8014a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8010156610 0x7f8010149c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.933+0000 7f8014a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80101edc30 con 0x7f80100c5060 2024-09-13T16:45:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.933+0000 7f8014a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80101ee4d0 con 0x7f80100c5440 2024-09-13T16:45:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.933+0000 7f8014a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80101eff50 con 0x7f8010156610 2024-09-13T16:45:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.933+0000 7f800f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100c5060 0x7f80101ef590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.933+0000 7f800ea00640 1 -- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80100c5440 msgr2=0x7f80101ee960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.191:3300/0 2024-09-13T16:45:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.933+0000 7f800f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100c5060 0x7f80101ef590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44220/0 (socket says 172.21.15.28:44220) 2024-09-13T16:45:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.933+0000 7f800f400640 1 -- 172.21.15.28:0/2326502532 learned_addr learned my addr 172.21.15.28:0/2326502532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.933+0000 7f800ea00640 1 --2- 172.21.15.28:0/2326502532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80100c5440 0x7f80101ee960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.933+0000 7f800fe00640 1 --2- 172.21.15.28:0/2326502532 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8010156610 0x7f8010149c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.934+0000 7f800f400640 1 -- 172.21.15.28:0/2326502532 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8010156610 msgr2=0x7f8010149c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.934+0000 7f800f400640 1 --2- 172.21.15.28:0/2326502532 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8010156610 0x7f8010149c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.934+0000 7f800f400640 1 -- 172.21.15.28:0/2326502532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80100c5440 msgr2=0x7f80101ee960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:45:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.934+0000 7f800f400640 1 --2- 172.21.15.28:0/2326502532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80100c5440 0x7f80101ee960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.934+0000 7f800f400640 1 -- 172.21.15.28:0/2326502532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8008009a70 con 0x7f80100c5060 2024-09-13T16:45:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.934+0000 7f800f400640 1 --2- 172.21.15.28:0/2326502532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100c5060 0x7f80101ef590 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f800000c4d0 tx=0x7f800000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.934+0000 7f8007400640 1 -- 172.21.15.28:0/2326502532 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8000005ce0 con 0x7f80100c5060 2024-09-13T16:45:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.934+0000 7f8007400640 1 -- 172.21.15.28:0/2326502532 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8000005e40 con 0x7f80100c5060 2024-09-13T16:45:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.934+0000 7f8007400640 1 -- 172.21.15.28:0/2326502532 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8000007270 con 0x7f80100c5060 2024-09-13T16:45:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.934+0000 7f8014a00640 1 -- 172.21.15.28:0/2326502532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80101ede80 con 0x7f80100c5060 2024-09-13T16:45:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.935+0000 7f8014a00640 1 -- 172.21.15.28:0/2326502532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80101e9d20 con 0x7f80100c5060 2024-09-13T16:45:41.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.936+0000 7f8014a00640 1 -- 172.21.15.28:0/2326502532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80100c7d00 con 0x7f80100c5060 2024-09-13T16:45:41.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.937+0000 7f8007400640 1 -- 172.21.15.28:0/2326502532 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f80000073d0 con 0x7f80100c5060 2024-09-13T16:45:41.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.938+0000 7f8007400640 1 --2- 172.21.15.28:0/2326502532 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ff006abe0 0x7f7ff006d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:41.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.938+0000 7f8007400640 1 -- 172.21.15.28:0/2326502532 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8000088830 con 0x7f80100c5060 2024-09-13T16:45:41.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.938+0000 7f800ea00640 1 --2- 172.21.15.28:0/2326502532 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ff006abe0 0x7f7ff006d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:41.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.939+0000 7f800ea00640 1 --2- 172.21.15.28:0/2326502532 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ff006abe0 0x7f7ff006d0a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f801006a5c0 tx=0x7f7ffc006d80 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:41.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:41.947+0000 7f8007400640 1 -- 172.21.15.28:0/2326502532 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f80000574c0 con 0x7f80100c5060 2024-09-13T16:45:42.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:42.192+0000 7f8014a00640 1 -- 172.21.15.28:0/2326502532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8010156bd0 con 0x7f80100c5060 2024-09-13T16:45:42.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:42.192+0000 7f8007400640 1 -- 172.21.15.28:0/2326502532 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f8000017620 con 0x7f80100c5060 2024-09-13T16:45:42.194 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:45:42.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:42.195+0000 7f8014a00640 1 -- 172.21.15.28:0/2326502532 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ff006abe0 msgr2=0x7f7ff006d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:42.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:42.195+0000 7f8014a00640 1 --2- 172.21.15.28:0/2326502532 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ff006abe0 0x7f7ff006d0a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f801006a5c0 tx=0x7f7ffc006d80 comp rx=0 tx=0).stop 2024-09-13T16:45:42.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:42.195+0000 7f8014a00640 1 -- 172.21.15.28:0/2326502532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100c5060 msgr2=0x7f80101ef590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:42.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:42.195+0000 7f8014a00640 1 --2- 172.21.15.28:0/2326502532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100c5060 0x7f80101ef590 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f800000c4d0 tx=0x7f800000c990 comp rx=0 tx=0).stop 2024-09-13T16:45:42.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:42.195+0000 7f8014a00640 1 -- 172.21.15.28:0/2326502532 shutdown_connections 2024-09-13T16:45:42.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:42.195+0000 7f8014a00640 1 --2- 172.21.15.28:0/2326502532 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ff006abe0 0x7f7ff006d0a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:42.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:42.195+0000 7f8014a00640 1 --2- 172.21.15.28:0/2326502532 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8010156610 0x7f8010149c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:42.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:42.195+0000 7f8014a00640 1 --2- 172.21.15.28:0/2326502532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f80100c5440 0x7f80101ee960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:42.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:42.195+0000 7f8014a00640 1 --2- 172.21.15.28:0/2326502532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80100c5060 0x7f80101ef590 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:42.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:42.195+0000 7f8014a00640 1 -- 172.21.15.28:0/2326502532 >> 172.21.15.28:0/2326502532 conn(0x7f8010069e80 msgr2=0x7f8010156df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:42.196+0000 7f8014a00640 1 -- 172.21.15.28:0/2326502532 shutdown_connections 2024-09-13T16:45:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:42.196+0000 7f8014a00640 1 -- 172.21.15.28:0/2326502532 wait complete. 2024-09-13T16:45:45.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:45:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.372+0000 7fa387e00640 1 Processor -- start 2024-09-13T16:45:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.372+0000 7fa387e00640 1 -- start start 2024-09-13T16:45:45.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.373+0000 7fa387e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 0x7fa3800c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:45.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.373+0000 7fa387e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3800c42b0 con 0x7fa38014fa80 2024-09-13T16:45:45.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.373+0000 7fa387e00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3800c36e0 con 0x7fa3800c6350 2024-09-13T16:45:45.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.373+0000 7fa387e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3800c3980 con 0x7fa380072170 2024-09-13T16:45:45.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.373+0000 7fa385e00640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7fa3800c6350 legacy=0x7fa3801559d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-09-13T16:45:45.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.373+0000 7fa385400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa38014fa80 0x7fa3800c6ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44864/0 (socket says 172.21.15.28:44864) 2024-09-13T16:45:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.373+0000 7fa385400640 1 -- 172.21.15.28:0/1225289522 learned_addr learned my addr 172.21.15.28:0/1225289522 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:45:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.373+0000 7fa384a00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 0x7fa3800c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.374+0000 7fa384a00640 1 -- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 msgr2=0x7fa3800c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T16:45:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.374+0000 7fa384a00640 1 -- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 msgr2=0x7fa3800c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:45:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.374+0000 7fa384a00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 0x7fa3800c1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.374+0000 7fa384a00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 0x7fa3800c1960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:45.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.574+0000 7fa385e00640 1 -- 172.21.15.28:0/1225289522 >> v1:172.21.15.191:6789/0 conn(0x7fa3800c6350 legacy=0x7fa3801559d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-09-13T16:45:45.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.575+0000 7fa384a00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 0x7fa3800c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:45.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.575+0000 7fa384a00640 1 -- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 msgr2=0x7fa3800c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T16:45:45.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.575+0000 7fa384a00640 1 -- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 msgr2=0x7fa3800c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:45:45.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.575+0000 7fa384a00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 0x7fa3800c1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:45.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.575+0000 7fa384a00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 0x7fa3800c1960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T16:45:45.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.975+0000 7fa385e00640 1 -- 172.21.15.28:0/1225289522 >> v1:172.21.15.191:6789/0 conn(0x7fa3800c6350 legacy=0x7fa3801559d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-09-13T16:45:45.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.976+0000 7fa384a00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 0x7fa3800c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:45.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.976+0000 7fa384a00640 1 -- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 msgr2=0x7fa3800c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T16:45:45.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.976+0000 7fa384a00640 1 -- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 msgr2=0x7fa3800c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:45:45.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.976+0000 7fa384a00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 0x7fa3800c1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:45.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:45.976+0000 7fa384a00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 0x7fa3800c1960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T16:45:46.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:46.777+0000 7fa385e00640 1 -- 172.21.15.28:0/1225289522 >> v1:172.21.15.191:6789/0 conn(0x7fa3800c6350 legacy=0x7fa3801559d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-09-13T16:45:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:46.779+0000 7fa384a00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 0x7fa3800c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:46.779+0000 7fa384a00640 1 -- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 msgr2=0x7fa3800c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T16:45:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:46.779+0000 7fa384a00640 1 -- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 msgr2=0x7fa3800c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:45:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:46.779+0000 7fa384a00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 0x7fa3800c1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:46.779+0000 7fa384a00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 0x7fa3800c1960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T16:45:48.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.374+0000 7fa37ea00640 1 -- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 msgr2=0x7fa3800c1960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:45:48.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.374+0000 7fa37ea00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.28:3300/0 conn(0x7fa380072170 0x7fa3800c1960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:48.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.374+0000 7fa37ea00640 1 -- 172.21.15.28:0/1225289522 >> v1:172.21.15.191:6789/0 conn(0x7fa3800c6350 legacy=0x7fa3801559d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:45:48.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.374+0000 7fa37ea00640 1 -- 172.21.15.28:0/1225289522 >> v1:172.21.15.28:6789/0 conn(0x7fa38014fa80 legacy=0x7fa3800c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:48.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.374+0000 7fa37ea00640 1 --2- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 0x7fa360005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:48.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.374+0000 7fa37ea00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.191:3300/0 conn(0x7fa360006180 0x7fa360008640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:48.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.374+0000 7fa37ea00640 1 -- 172.21.15.28:0/1225289522 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3800c3980 con 0x7fa360006180 2024-09-13T16:45:48.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.374+0000 7fa37ea00640 1 -- 172.21.15.28:0/1225289522 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa360008cf0 con 0x7fa3600036a0 2024-09-13T16:45:48.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.374+0000 7fa384a00640 1 -- 172.21.15.28:0/1225289522 >> v2:172.21.15.191:3300/0 conn(0x7fa360006180 msgr2=0x7fa360008640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3300/0 2024-09-13T16:45:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.374+0000 7fa384a00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.191:3300/0 conn(0x7fa360006180 0x7fa360008640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.375+0000 7fa385400640 1 --2- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 0x7fa360005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.375+0000 7fa385400640 1 -- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 msgr2=0x7fa360005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:45:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.375+0000 7fa385400640 1 -- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 msgr2=0x7fa360005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:45:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.375+0000 7fa385400640 1 --2- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 0x7fa360005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.375+0000 7fa385400640 1 --2- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 0x7fa360005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:48.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.575+0000 7fa384a00640 1 -- 172.21.15.28:0/1225289522 >> v2:172.21.15.191:3300/0 conn(0x7fa360006180 msgr2=0x7fa360008640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3300/0 2024-09-13T16:45:48.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.575+0000 7fa384a00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.191:3300/0 conn(0x7fa360006180 0x7fa360008640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T16:45:48.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.576+0000 7fa385400640 1 --2- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 0x7fa360005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.576+0000 7fa385400640 1 -- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 msgr2=0x7fa360005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:45:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.576+0000 7fa385400640 1 -- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 msgr2=0x7fa360005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:45:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.576+0000 7fa385400640 1 --2- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 0x7fa360005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.576+0000 7fa385400640 1 --2- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 0x7fa360005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T16:45:48.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.976+0000 7fa384a00640 1 -- 172.21.15.28:0/1225289522 >> v2:172.21.15.191:3300/0 conn(0x7fa360006180 msgr2=0x7fa360008640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3300/0 2024-09-13T16:45:48.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.976+0000 7fa384a00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.191:3300/0 conn(0x7fa360006180 0x7fa360008640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T16:45:48.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.977+0000 7fa385400640 1 --2- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 0x7fa360005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:48.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.977+0000 7fa385400640 1 -- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 msgr2=0x7fa360005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:45:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.977+0000 7fa385400640 1 -- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 msgr2=0x7fa360005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:45:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.977+0000 7fa385400640 1 --2- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 0x7fa360005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:45:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:48.977+0000 7fa385400640 1 --2- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 0x7fa360005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T16:45:49.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.777+0000 7fa384a00640 1 -- 172.21.15.28:0/1225289522 >> v2:172.21.15.191:3300/0 conn(0x7fa360006180 msgr2=0x7fa360008640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3300/0 2024-09-13T16:45:49.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.777+0000 7fa384a00640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.191:3300/0 conn(0x7fa360006180 0x7fa360008640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T16:45:49.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.778+0000 7fa385400640 1 --2- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 0x7fa360005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:49.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.779+0000 7fa385400640 1 -- 172.21.15.28:0/1225289522 >> v2:172.21.15.191:3300/0 conn(0x7fa360006180 msgr2=0x7fa360008640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:45:49.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.779+0000 7fa385400640 1 --2- 172.21.15.28:0/1225289522 >> v2:172.21.15.191:3300/0 conn(0x7fa360006180 0x7fa360008640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:49.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.779+0000 7fa385400640 1 -- 172.21.15.28:0/1225289522 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3800c3c60 con 0x7fa3600036a0 2024-09-13T16:45:49.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.780+0000 7fa385400640 1 --2- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 0x7fa360005af0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa374007d20 tx=0x7fa37402a7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b9f5414a4168210 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:49.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.780+0000 7fa37fe00640 1 -- 172.21.15.28:0/1225289522 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa374033410 con 0x7fa3600036a0 2024-09-13T16:45:49.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.780+0000 7fa37fe00640 1 -- 172.21.15.28:0/1225289522 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37402d030 con 0x7fa3600036a0 2024-09-13T16:45:49.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.780+0000 7fa37fe00640 1 -- 172.21.15.28:0/1225289522 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa374033770 con 0x7fa3600036a0 2024-09-13T16:45:49.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.781+0000 7fa387e00640 1 -- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 msgr2=0x7fa360005af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:49.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.781+0000 7fa387e00640 1 --2- 172.21.15.28:0/1225289522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 0x7fa360005af0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa374007d20 tx=0x7fa37402a7d0 comp rx=0 tx=0).stop 2024-09-13T16:45:49.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.781+0000 7fa385e00640 1 -- 172.21.15.28:0/1225289522 reap_dead start 2024-09-13T16:45:49.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.781+0000 7fa387e00640 1 -- 172.21.15.28:0/1225289522 shutdown_connections 2024-09-13T16:45:49.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.781+0000 7fa387e00640 1 -- 172.21.15.28:0/1225289522 >> 172.21.15.28:0/1225289522 conn(0x7fa380069e80 msgr2=0x7fa38006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:49.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.781+0000 7fa387e00640 1 -- 172.21.15.28:0/1225289522 shutdown_connections 2024-09-13T16:45:49.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.781+0000 7fa387e00640 1 -- 172.21.15.28:0/1225289522 wait complete. 2024-09-13T16:45:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.783+0000 7fa387e00640 1 Processor -- start 2024-09-13T16:45:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.783+0000 7fa387e00640 1 -- start start 2024-09-13T16:45:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.784+0000 7fa387e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 0x7fa380155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.784+0000 7fa387e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa38015afe0 0x7fa3801f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.784+0000 7fa387e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3801f2580 0x7fa380148d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.784+0000 7fa387e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38015b6c0 con 0x7fa38015afe0 2024-09-13T16:45:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.784+0000 7fa387e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa380151550 con 0x7fa3801f2580 2024-09-13T16:45:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.784+0000 7fa387e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa380150b10 con 0x7fa3600036a0 2024-09-13T16:45:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.784+0000 7fa385e00640 1 -- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3801f2580 msgr2=0x7fa380148d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.191:3300/0 2024-09-13T16:45:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.784+0000 7fa385e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3801f2580 0x7fa380148d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.784+0000 7fa384a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa38015afe0 0x7fa3801f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.784+0000 7fa384a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa38015afe0 0x7fa3801f2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42396/0 (socket says 172.21.15.28:42396) 2024-09-13T16:45:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.784+0000 7fa385400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 0x7fa380155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.784+0000 7fa384a00640 1 -- 172.21.15.28:0/3877448660 learned_addr learned my addr 172.21.15.28:0/3877448660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.784+0000 7fa384a00640 1 -- 172.21.15.28:0/3877448660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 msgr2=0x7fa380155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.784+0000 7fa384a00640 1 --2- 172.21.15.28:0/3877448660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 0x7fa380155000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.784+0000 7fa384a00640 1 -- 172.21.15.28:0/3877448660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3801f2580 msgr2=0x7fa380148d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:45:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.784+0000 7fa384a00640 1 --2- 172.21.15.28:0/3877448660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3801f2580 0x7fa380148d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.785+0000 7fa384a00640 1 -- 172.21.15.28:0/3877448660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37400f040 con 0x7fa38015afe0 2024-09-13T16:45:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.785+0000 7fa384a00640 1 --2- 172.21.15.28:0/3877448660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa38015afe0 0x7fa3801f2040 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa36c006ce0 tx=0x7fa36c013a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.785+0000 7fa37d600640 1 -- 172.21.15.28:0/3877448660 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c015690 con 0x7fa38015afe0 2024-09-13T16:45:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.785+0000 7fa37d600640 1 -- 172.21.15.28:0/3877448660 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36c015810 con 0x7fa38015afe0 2024-09-13T16:45:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.785+0000 7fa387e00640 1 -- 172.21.15.28:0/3877448660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3801f1bb0 con 0x7fa38015afe0 2024-09-13T16:45:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.785+0000 7fa37d600640 1 -- 172.21.15.28:0/3877448660 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c015bc0 con 0x7fa38015afe0 2024-09-13T16:45:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.785+0000 7fa387e00640 1 -- 172.21.15.28:0/3877448660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa380150270 con 0x7fa38015afe0 2024-09-13T16:45:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.787+0000 7fa387e00640 1 -- 172.21.15.28:0/3877448660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3801f1d40 con 0x7fa38015afe0 2024-09-13T16:45:49.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.788+0000 7fa37d600640 1 -- 172.21.15.28:0/3877448660 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa36c015d20 con 0x7fa38015afe0 2024-09-13T16:45:49.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.789+0000 7fa37d600640 1 --2- 172.21.15.28:0/3877448660 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa36006b8e0 0x7fa36006dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:49.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.789+0000 7fa37d600640 1 -- 172.21.15.28:0/3877448660 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa36c0898d0 con 0x7fa38015afe0 2024-09-13T16:45:49.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.789+0000 7fa385400640 1 --2- 172.21.15.28:0/3877448660 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa36006b8e0 0x7fa36006dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:49.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.790+0000 7fa385400640 1 --2- 172.21.15.28:0/3877448660 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa36006b8e0 0x7fa36006dda0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa374035000 tx=0x7fa374030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:49.793+0000 7fa37d600640 1 -- 172.21.15.28:0/3877448660 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa36c054a40 con 0x7fa38015afe0 2024-09-13T16:45:49.846 INFO:tasks.workunit.client.0.smithi028.stderr:Updating files: 60% (7700/12697) Updating files: 61% (7746/12697) Updating files: 62% (7873/12697) Updating files: 63% (8000/12697) Updating files: 64% (8127/12697) Updating files: 65% (8254/12697) Updating files: 66% (8381/12697) Updating files: 67% (8507/12697) Updating files: 68% (8634/12697) Updating files: 69% (8761/12697) Updating files: 70% (8888/12697) Updating files: 71% (9015/12697) Updating files: 72% (9142/12697) Updating files: 73% (9269/12697) Updating files: 74% (9396/12697) Updating files: 75% (9523/12697) Updating files: 76% (9650/12697) Updating files: 77% (9777/12697) Updating files: 78% (9904/12697) Updating files: 79% (10031/12697) Updating files: 80% (10158/12697) Updating files: 81% (10285/12697) Updating files: 82% (10412/12697) Updating files: 83% (10539/12697) Updating files: 84% (10666/12697) Updating files: 85% (10793/12697) Updating files: 86% (10920/12697) Updating files: 87% (11047/12697) Updating files: 88% (11174/12697) Updating files: 89% (11301/12697) Updating files: 90% (11428/12697) Updating files: 91% (11555/12697) Updating files: 92% (11682/12697) Updating files: 93% (11809/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 99% (12654/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-09-13T16:45:50.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.046+0000 7fa387e00640 1 -- 172.21.15.28:0/3877448660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa3800c1960 con 0x7fa38015afe0 2024-09-13T16:45:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.046+0000 7fa37d600640 1 -- 172.21.15.28:0/3877448660 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7fa36c024080 con 0x7fa38015afe0 2024-09-13T16:45:50.048 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":14,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:45:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.048+0000 7fa387e00640 1 -- 172.21.15.28:0/3877448660 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa36006b8e0 msgr2=0x7fa36006dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.048+0000 7fa387e00640 1 --2- 172.21.15.28:0/3877448660 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa36006b8e0 0x7fa36006dda0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa374035000 tx=0x7fa374030000 comp rx=0 tx=0).stop 2024-09-13T16:45:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.048+0000 7fa387e00640 1 -- 172.21.15.28:0/3877448660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa38015afe0 msgr2=0x7fa3801f2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.048+0000 7fa387e00640 1 --2- 172.21.15.28:0/3877448660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa38015afe0 0x7fa3801f2040 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa36c006ce0 tx=0x7fa36c013a20 comp rx=0 tx=0).stop 2024-09-13T16:45:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.049+0000 7fa387e00640 1 -- 172.21.15.28:0/3877448660 shutdown_connections 2024-09-13T16:45:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.049+0000 7fa387e00640 1 --2- 172.21.15.28:0/3877448660 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa36006b8e0 0x7fa36006dda0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.049+0000 7fa387e00640 1 --2- 172.21.15.28:0/3877448660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3801f2580 0x7fa380148d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.049+0000 7fa387e00640 1 --2- 172.21.15.28:0/3877448660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa38015afe0 0x7fa3801f2040 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.049+0000 7fa387e00640 1 --2- 172.21.15.28:0/3877448660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3600036a0 0x7fa380155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.049+0000 7fa387e00640 1 -- 172.21.15.28:0/3877448660 >> 172.21.15.28:0/3877448660 conn(0x7fa380069e80 msgr2=0x7fa380156090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.049+0000 7fa387e00640 1 -- 172.21.15.28:0/3877448660 shutdown_connections 2024-09-13T16:45:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.049+0000 7fa387e00640 1 -- 172.21.15.28:0/3877448660 wait complete. 2024-09-13T16:45:50.066 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-13T16:45:50.066 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-13T16:45:50.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T16:45:50.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.222+0000 7fa116000640 1 Processor -- start 2024-09-13T16:45:50.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.222+0000 7fa116000640 1 -- start start 2024-09-13T16:45:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.223+0000 7fa116000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa11015a440 0x7fa1100c5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.223+0000 7fa116000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa11006dcb0 0x7fa1101504e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.223+0000 7fa116000640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1100c1d80 con 0x7fa11015a060 2024-09-13T16:45:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.223+0000 7fa116000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11015a920 con 0x7fa11006dcb0 2024-09-13T16:45:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.223+0000 7fa116000640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11015ac00 con 0x7fa11015a440 2024-09-13T16:45:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.223+0000 7fa10ea00640 1 -- >> v2:172.21.15.191:3300/0 conn(0x7fa11015a440 msgr2=0x7fa1100c5500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.191:3300/0 2024-09-13T16:45:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.223+0000 7fa10f400640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7fa11015a060 legacy=0x7fa11006d5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-09-13T16:45:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.224+0000 7fa10ea00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa11015a440 0x7fa1100c5500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.224+0000 7fa10fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa11006dcb0 0x7fa1101504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.224+0000 7fa10fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa11006dcb0 0x7fa1101504e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42410/0 (socket says 172.21.15.28:42410) 2024-09-13T16:45:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.224+0000 7fa10fe00640 1 -- 172.21.15.28:0/1690654451 learned_addr learned my addr 172.21.15.28:0/1690654451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:50.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.224+0000 7fa10fe00640 1 -- 172.21.15.28:0/1690654451 >> v2:172.21.15.191:3300/0 conn(0x7fa11015a440 msgr2=0x7fa1100c5500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:45:50.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.224+0000 7fa10fe00640 1 --2- 172.21.15.28:0/1690654451 >> v2:172.21.15.191:3300/0 conn(0x7fa11015a440 0x7fa1100c5500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.224+0000 7fa10fe00640 1 -- 172.21.15.28:0/1690654451 >> v1:172.21.15.191:6789/0 conn(0x7fa11015a060 legacy=0x7fa11006d5a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:45:50.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.224+0000 7fa10fe00640 1 -- 172.21.15.28:0/1690654451 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1100c2d50 con 0x7fa11006dcb0 2024-09-13T16:45:50.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.226+0000 7fa10fe00640 1 --2- 172.21.15.28:0/1690654451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa11006dcb0 0x7fa1101504e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa1100c47e0 tx=0x7fa10802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ec688ce3d6b9f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:50.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.226+0000 7fa10e000640 1 -- 172.21.15.28:0/1690654451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa108031600 con 0x7fa11006dcb0 2024-09-13T16:45:50.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.226+0000 7fa10e000640 1 -- 172.21.15.28:0/1690654451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10802d050 con 0x7fa11006dcb0 2024-09-13T16:45:50.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.226+0000 7fa10e000640 1 -- 172.21.15.28:0/1690654451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa108031980 con 0x7fa11006dcb0 2024-09-13T16:45:50.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.226+0000 7fa116000640 1 -- 172.21.15.28:0/1690654451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa11006dcb0 msgr2=0x7fa1101504e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:50.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.226+0000 7fa116000640 1 --2- 172.21.15.28:0/1690654451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa11006dcb0 0x7fa1101504e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa1100c47e0 tx=0x7fa10802a710 comp rx=0 tx=0).stop 2024-09-13T16:45:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.227+0000 7fa116000640 1 -- 172.21.15.28:0/1690654451 shutdown_connections 2024-09-13T16:45:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.227+0000 7fa116000640 1 --2- 172.21.15.28:0/1690654451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa11006dcb0 0x7fa1101504e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.227+0000 7fa116000640 1 --2- 172.21.15.28:0/1690654451 >> v2:172.21.15.191:3300/0 conn(0x7fa11015a440 0x7fa1100c5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.227+0000 7fa116000640 1 -- 172.21.15.28:0/1690654451 >> 172.21.15.28:0/1690654451 conn(0x7fa1100ca6b0 msgr2=0x7fa1100caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.227+0000 7fa116000640 1 -- 172.21.15.28:0/1690654451 shutdown_connections 2024-09-13T16:45:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.227+0000 7fa116000640 1 -- 172.21.15.28:0/1690654451 wait complete. 2024-09-13T16:45:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.228+0000 7fa116000640 1 Processor -- start 2024-09-13T16:45:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.228+0000 7fa116000640 1 -- start start 2024-09-13T16:45:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.229+0000 7fa116000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa11006dcb0 0x7fa110158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.229+0000 7fa116000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa11015a060 0x7fa1101593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.229+0000 7fa116000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa11015a440 0x7fa1101e1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.229+0000 7fa116000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11014af90 con 0x7fa11006dcb0 2024-09-13T16:45:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.229+0000 7fa116000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1101ef960 con 0x7fa11015a440 2024-09-13T16:45:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.229+0000 7fa116000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa11014acf0 con 0x7fa11015a060 2024-09-13T16:45:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.229+0000 7fa10fe00640 1 -- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa11015a440 msgr2=0x7fa1101e1ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.191:3300/0 2024-09-13T16:45:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.229+0000 7fa10f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa11006dcb0 0x7fa110158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.230+0000 7fa10fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa11015a440 0x7fa1101e1ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.230+0000 7fa10f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa11006dcb0 0x7fa110158e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42414/0 (socket says 172.21.15.28:42414) 2024-09-13T16:45:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.230+0000 7fa10f400640 1 -- 172.21.15.28:0/4083453152 learned_addr learned my addr 172.21.15.28:0/4083453152 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.230+0000 7fa10ea00640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa11015a060 0x7fa1101593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.230+0000 7fa10f400640 1 -- 172.21.15.28:0/4083453152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa11015a060 msgr2=0x7fa1101593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.230+0000 7fa10f400640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa11015a060 0x7fa1101593b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.230+0000 7fa10f400640 1 -- 172.21.15.28:0/4083453152 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa11015a440 msgr2=0x7fa1101e1ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:45:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.230+0000 7fa10f400640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa11015a440 0x7fa1101e1ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.230+0000 7fa10f400640 1 -- 172.21.15.28:0/4083453152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa108009b30 con 0x7fa11006dcb0 2024-09-13T16:45:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.230+0000 7fa10f400640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa11006dcb0 0x7fa110158e70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa10000b550 tx=0x7fa10000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.230+0000 7fa107400640 1 -- 172.21.15.28:0/4083453152 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa100003cf0 con 0x7fa11006dcb0 2024-09-13T16:45:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.230+0000 7fa107400640 1 -- 172.21.15.28:0/4083453152 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa100003e70 con 0x7fa11006dcb0 2024-09-13T16:45:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.230+0000 7fa107400640 1 -- 172.21.15.28:0/4083453152 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa100005290 con 0x7fa11006dcb0 2024-09-13T16:45:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.231+0000 7fa116000640 1 -- 172.21.15.28:0/4083453152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa11014ebd0 con 0x7fa11006dcb0 2024-09-13T16:45:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.231+0000 7fa116000640 1 -- 172.21.15.28:0/4083453152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11014b890 con 0x7fa11006dcb0 2024-09-13T16:45:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.232+0000 7fa116000640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1101e2010 0x7fa1101f1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.232+0000 7fa116000640 1 -- 172.21.15.28:0/4083453152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa110066c20 con 0x7fa1101e2010 2024-09-13T16:45:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.232+0000 7fa10ea00640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1101e2010 0x7fa1101f1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.232+0000 7fa10ea00640 1 -- 172.21.15.28:0/4083453152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1101ef470 con 0x7fa1101e2010 2024-09-13T16:45:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.233+0000 7fa10ea00640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1101e2010 0x7fa1101f1f60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa110067150 tx=0x7fa0fc009ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.233+0000 7fa107400640 1 -- 172.21.15.28:0/4083453152 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa1000053f0 con 0x7fa11006dcb0 2024-09-13T16:45:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.234+0000 7fa107400640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa0f006abb0 0x7fa0f006d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.234+0000 7fa107400640 1 -- 172.21.15.28:0/4083453152 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa100089320 con 0x7fa11006dcb0 2024-09-13T16:45:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.234+0000 7fa107400640 1 -- 172.21.15.28:0/4083453152 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc012600 con 0x7fa1101e2010 2024-09-13T16:45:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.234+0000 7fa107400640 1 -- 172.21.15.28:0/4083453152 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fa0fc012760 con 0x7fa1101e2010 2024-09-13T16:45:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.234+0000 7fa107400640 1 -- 172.21.15.28:0/4083453152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1101e2010 msgr2=0x7fa1101f1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.234+0000 7fa107400640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1101e2010 0x7fa1101f1f60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa110067150 tx=0x7fa0fc009ae0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.234+0000 7fa107400640 1 -- 172.21.15.28:0/4083453152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1101e2010 msgr2=0x7fa1101f1f60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:45:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.234+0000 7fa107400640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1101e2010 0x7fa1101f1f60 secure :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa110067150 tx=0x7fa0fc009ae0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.234+0000 7fa10fe00640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa0f006abb0 0x7fa0f006d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.235+0000 7fa10fe00640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa0f006abb0 0x7fa0f006d070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa108033000 tx=0x7fa108009000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.241+0000 7fa116000640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1101f24a0 0x7fa1101f28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.241+0000 7fa116000640 1 -- 172.21.15.28:0/4083453152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1100674a0 con 0x7fa1101f24a0 2024-09-13T16:45:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.241+0000 7fa10ea00640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1101f24a0 0x7fa1101f28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.241+0000 7fa10ea00640 1 -- 172.21.15.28:0/4083453152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa1100c5220 con 0x7fa1101f24a0 2024-09-13T16:45:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.242+0000 7fa10ea00640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1101f24a0 0x7fa1101f28b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa11014c5f0 tx=0x7fa0fc0059c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.242+0000 7fa107400640 1 -- 172.21.15.28:0/4083453152 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc022070 con 0x7fa1101f24a0 2024-09-13T16:45:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.242+0000 7fa107400640 1 -- 172.21.15.28:0/4083453152 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+5089 (secure 0 0 0) 0x7fa0fc006310 con 0x7fa1101f24a0 2024-09-13T16:45:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.242+0000 7fa107400640 1 -- 172.21.15.28:0/4083453152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1101f24a0 msgr2=0x7fa1101f28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.242+0000 7fa107400640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1101f24a0 0x7fa1101f28b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa11014c5f0 tx=0x7fa0fc0059c0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.242+0000 7fa107400640 1 -- 172.21.15.28:0/4083453152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1101f24a0 msgr2=0x7fa1101f28b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:45:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.242+0000 7fa107400640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1101f24a0 0x7fa1101f28b0 secure :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa11014c5f0 tx=0x7fa0fc0059c0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.244 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:45:50.245 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:45:50.245 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:45:50.245 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "leader", 2024-09-13T16:45:50.245 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 14, 2024-09-13T16:45:50.245 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:45:50.245 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:45:50.245 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:45:50.245 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:50.245 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 0, 2024-09-13T16:45:50.245 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:45:50.245 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:45:50.246 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:45:50.246 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:50.246 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:50.246 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:50.246 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:50.246 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:50.246 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:50.246 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:50.246 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:50.246 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:50.246 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:50.246 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:50.246 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:45:50.247 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:45:50.247 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:50.247 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:50.247 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:50.247 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:50.247 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:50.247 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:50.247 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:50.247 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:50.247 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:50.247 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:50.247 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:50.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.248 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:45:50.248 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:45:50.248 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:45:50.248 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:45:50.248 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:45:50.248 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:45:50.248 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:45:50.248 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:45:50.248 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:45:50.248 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:45:50.248 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:45:50.248 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:45:50.248 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:45:50.249 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:45:50.249 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:45:50.249 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:45:50.249 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:45:50.249 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:50.249 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:50.249 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:50.249 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:50.249 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:50.249 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:50.249 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:50.249 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:50.250 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:50.250 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:50.250 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:50.250 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:45:50.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.250 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:45:50.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.250 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:45:50.250 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:45:50.250 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:50.250 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:50.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:50.251 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:45:50.251 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:50.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.251 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:50.251 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:45:50.251 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:50.251 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:50.251 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:50.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.251 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:45:50.251 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:45:50.252 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:50.252 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:50.252 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:50.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.252 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:45:50.252 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:45:50.252 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:50.252 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:50.252 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:50.252 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:45:50.252 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:50.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.253 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.253 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:50.253 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:45:50.253 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:50.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:50.253 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:50.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.253 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:45:50.253 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:45:50.253 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:50.253 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:50.253 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:50.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.254 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:45:50.254 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:45:50.254 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:50.254 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:50.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:50.254 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:45:50.254 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:50.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:50.255 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:45:50.255 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:50.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:50.255 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:50.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.255 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:45:50.255 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:45:50.255 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:50.255 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:50.255 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:50.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:50.255 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:50.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.256 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:45:50.256 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:45:50.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.256 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:50.256 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:50.256 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:45:50.256 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:50.256 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:50.256 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:45:50.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.256 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:50.256 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:50.256 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:45:50.257 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:50.257 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:50.257 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:45:50.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.257 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:50.257 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:50.257 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 5 2024-09-13T16:45:50.257 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:50.257 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:50.257 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:45:50.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.257 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x2f018fb87aa4aafe", 2024-09-13T16:45:50.257 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "luminous", 2024-09-13T16:45:50.258 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:45:50.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.258 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:50.258 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:50.258 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 5 2024-09-13T16:45:50.258 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:50.258 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:50.258 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": [ 2024-09-13T16:45:50.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.258 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:50.258 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:50.258 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:45:50.259 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:50.259 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:50.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.259 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:45:50.259 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:45:50.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.243+0000 7fa116000640 1 -- 172.21.15.28:0/4083453152 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa0f006abb0 msgr2=0x7fa0f006d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:50.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.243+0000 7fa116000640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa0f006abb0 0x7fa0f006d070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa108033000 tx=0x7fa108009000 comp rx=0 tx=0).stop 2024-09-13T16:45:50.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.243+0000 7fa116000640 1 -- 172.21.15.28:0/4083453152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa11006dcb0 msgr2=0x7fa110158e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:50.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.243+0000 7fa116000640 1 --2- 172.21.15.28:0/4083453152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa11006dcb0 0x7fa110158e70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa10000b550 tx=0x7fa10000ba10 comp rx=0 tx=0).stop 2024-09-13T16:45:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.243+0000 7fa10fe00640 1 -- 172.21.15.28:0/4083453152 reap_dead start 2024-09-13T16:45:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.243+0000 7fa10fe00640 1 -- 172.21.15.28:0/4083453152 reap_dead start 2024-09-13T16:45:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.243+0000 7fa116000640 1 -- 172.21.15.28:0/4083453152 shutdown_connections 2024-09-13T16:45:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.243+0000 7fa116000640 1 -- 172.21.15.28:0/4083453152 >> 172.21.15.28:0/4083453152 conn(0x7fa1100ca6b0 msgr2=0x7fa1100caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.243+0000 7fa116000640 1 -- 172.21.15.28:0/4083453152 shutdown_connections 2024-09-13T16:45:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.243+0000 7fa116000640 1 -- 172.21.15.28:0/4083453152 wait complete. 2024-09-13T16:45:50.261 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-13T16:45:50.261 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-13T16:45:50.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T16:45:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.416+0000 7f8609600640 1 Processor -- start 2024-09-13T16:45:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.416+0000 7f8609600640 1 -- start start 2024-09-13T16:45:50.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.418+0000 7f8609600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f860415a060 0x7f86040c5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:50.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.418+0000 7f8609600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f860415a440 0x7f860406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:50.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.418+0000 7f8609600640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86040c1d80 con 0x7f86040c62a0 2024-09-13T16:45:50.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.418+0000 7f8609600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860415a920 con 0x7f860415a060 2024-09-13T16:45:50.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.418+0000 7f8609600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860415ac00 con 0x7f860415a440 2024-09-13T16:45:50.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.418+0000 7f8603400640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f86040c62a0 legacy=0x7f86041504e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-09-13T16:45:50.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.418+0000 7f8602a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f860415a060 0x7f86040c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:50.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.418+0000 7f8602a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f860415a060 0x7f86040c5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42444/0 (socket says 172.21.15.28:42444) 2024-09-13T16:45:50.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.418+0000 7f85fbe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f860415a440 0x7f860406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:50.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.418+0000 7f8602a00640 1 -- 172.21.15.28:0/1196845683 learned_addr learned my addr 172.21.15.28:0/1196845683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.418+0000 7f8602a00640 1 -- 172.21.15.28:0/1196845683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f860415a440 msgr2=0x7f860406d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:45:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.419+0000 7f8602a00640 1 --2- 172.21.15.28:0/1196845683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f860415a440 0x7f860406d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.419+0000 7f8602a00640 1 -- 172.21.15.28:0/1196845683 >> v1:172.21.15.191:6789/0 conn(0x7f86040c62a0 legacy=0x7f86041504e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:45:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.419+0000 7f8602a00640 1 -- 172.21.15.28:0/1196845683 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86040c2d50 con 0x7f860415a060 2024-09-13T16:45:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.419+0000 7f85fbe00640 1 --2- 172.21.15.28:0/1196845683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f860415a440 0x7f860406d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:45:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.420+0000 7f8602a00640 1 --2- 172.21.15.28:0/1196845683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f860415a060 0x7f86040c5500 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f85f4009870 tx=0x7f85f402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22e7333b660ca727 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:50.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.420+0000 7f8602000640 1 -- 172.21.15.28:0/1196845683 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f4032440 con 0x7f860415a060 2024-09-13T16:45:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.420+0000 7f8602000640 1 -- 172.21.15.28:0/1196845683 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f40325a0 con 0x7f860415a060 2024-09-13T16:45:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.420+0000 7f8602000640 1 -- 172.21.15.28:0/1196845683 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f4032930 con 0x7f860415a060 2024-09-13T16:45:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.421+0000 7f8609600640 1 -- 172.21.15.28:0/1196845683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f860415a060 msgr2=0x7f86040c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.421+0000 7f8609600640 1 --2- 172.21.15.28:0/1196845683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f860415a060 0x7f86040c5500 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f85f4009870 tx=0x7f85f402a210 comp rx=0 tx=0).stop 2024-09-13T16:45:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.421+0000 7f8609600640 1 -- 172.21.15.28:0/1196845683 shutdown_connections 2024-09-13T16:45:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.421+0000 7f8609600640 1 --2- 172.21.15.28:0/1196845683 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f860415a440 0x7f860406d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.421+0000 7f8609600640 1 --2- 172.21.15.28:0/1196845683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f860415a060 0x7f86040c5500 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.421+0000 7f8609600640 1 -- 172.21.15.28:0/1196845683 >> 172.21.15.28:0/1196845683 conn(0x7f86040ca6b0 msgr2=0x7f86040caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.421+0000 7f8609600640 1 -- 172.21.15.28:0/1196845683 shutdown_connections 2024-09-13T16:45:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.421+0000 7f8609600640 1 -- 172.21.15.28:0/1196845683 wait complete. 2024-09-13T16:45:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.422+0000 7f8609600640 1 Processor -- start 2024-09-13T16:45:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.423+0000 7f8609600640 1 -- start start 2024-09-13T16:45:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.423+0000 7f8609600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86040c62a0 0x7f86041522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.423+0000 7f8609600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f860415a060 0x7f8604159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.424+0000 7f8609600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f860415a440 0x7f86041e0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.424+0000 7f8609600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8604153a60 con 0x7f860415a440 2024-09-13T16:45:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.424+0000 7f8609600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8604148ed0 con 0x7f86040c62a0 2024-09-13T16:45:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.424+0000 7f8609600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86041513e0 con 0x7f860415a060 2024-09-13T16:45:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.424+0000 7f8602a00640 1 -- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86040c62a0 msgr2=0x7f86041522a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.191:3300/0 2024-09-13T16:45:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.424+0000 7f8602a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86040c62a0 0x7f86041522a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:45:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.424+0000 7f8603400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f860415a440 0x7f86041e0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.424+0000 7f85fbe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f860415a060 0x7f8604159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.424+0000 7f8603400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f860415a440 0x7f86041e0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42450/0 (socket says 172.21.15.28:42450) 2024-09-13T16:45:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.424+0000 7f8603400640 1 -- 172.21.15.28:0/1260293241 learned_addr learned my addr 172.21.15.28:0/1260293241 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:45:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.424+0000 7f85fbe00640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f860415a060 0x7f8604159530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:41774/0 (socket says 172.21.15.28:41774) 2024-09-13T16:45:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.424+0000 7f8603400640 1 -- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f860415a060 msgr2=0x7f8604159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.424+0000 7f8603400640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f860415a060 0x7f8604159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.424+0000 7f8603400640 1 -- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86040c62a0 msgr2=0x7f86041522a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:45:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.424+0000 7f8603400640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f86040c62a0 0x7f86041522a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.424+0000 7f8603400640 1 -- 172.21.15.28:0/1260293241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec009400 con 0x7f860415a440 2024-09-13T16:45:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.424+0000 7f85fbe00640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f860415a060 0x7f8604159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:45:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.425+0000 7f8603400640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f860415a440 0x7f86041e0100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00b9c0 tx=0x7f85fc00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.425+0000 7f85faa00640 1 -- 172.21.15.28:0/1260293241 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc00cb20 con 0x7f860415a440 2024-09-13T16:45:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.425+0000 7f85faa00640 1 -- 172.21.15.28:0/1260293241 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85fc00cc80 con 0x7f860415a440 2024-09-13T16:45:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.425+0000 7f85faa00640 1 -- 172.21.15.28:0/1260293241 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc015690 con 0x7f860415a440 2024-09-13T16:45:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.425+0000 7f8609600640 1 -- 172.21.15.28:0/1260293241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f40094d0 con 0x7f860415a440 2024-09-13T16:45:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.425+0000 7f8609600640 1 -- 172.21.15.28:0/1260293241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860414da30 con 0x7f860415a440 2024-09-13T16:45:50.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.426+0000 7f8609600640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8604151e30 0x7f86041e0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:50.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.426+0000 7f8609600640 1 -- 172.21.15.28:0/1260293241 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8604066c20 con 0x7f8604151e30 2024-09-13T16:45:50.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.427+0000 7f8602a00640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8604151e30 0x7f86041e0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:50.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.427+0000 7f8602a00640 1 -- 172.21.15.28:0/1260293241 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8604151270 con 0x7f8604151e30 2024-09-13T16:45:50.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.427+0000 7f8602a00640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8604151e30 0x7f86041e0640 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f86040c8060 tx=0x7f85f40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:50.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.427+0000 7f85faa00640 1 -- 172.21.15.28:0/1260293241 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f4032440 con 0x7f8604151e30 2024-09-13T16:45:50.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.428+0000 7f85faa00640 1 -- 172.21.15.28:0/1260293241 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f85f4004220 con 0x7f8604151e30 2024-09-13T16:45:50.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.428+0000 7f85faa00640 1 -- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8604151e30 msgr2=0x7f86041e0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:50.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.428+0000 7f85faa00640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8604151e30 0x7f86041e0640 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f86040c8060 tx=0x7f85f40046c0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.428+0000 7f85faa00640 1 -- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8604151e30 msgr2=0x7f86041e0640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:45:50.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.428+0000 7f85faa00640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8604151e30 0x7f86041e0640 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:45:50.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.428+0000 7f85faa00640 1 -- 172.21.15.28:0/1260293241 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f85fc012070 con 0x7f860415a440 2024-09-13T16:45:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.429+0000 7f85faa00640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f85e006a940 0x7f85e006ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.429+0000 7f85faa00640 1 -- 172.21.15.28:0/1260293241 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f85fc0892d0 con 0x7f860415a440 2024-09-13T16:45:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.429+0000 7f8602a00640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f85e006a940 0x7f85e006ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.430+0000 7f8602a00640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f85e006a940 0x7f85e006ce00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f860406ba20 tx=0x7f85f403d000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.435+0000 7f8609600640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8604153d70 0x7f8604151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:45:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.435+0000 7f8609600640 1 -- 172.21.15.28:0/1260293241 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86040c5220 con 0x7f8604153d70 2024-09-13T16:45:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.435+0000 7f85fbe00640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8604153d70 0x7f8604151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:45:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.436+0000 7f85fbe00640 1 -- 172.21.15.28:0/1260293241 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f86040c3310 con 0x7f8604153d70 2024-09-13T16:45:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.436+0000 7f85fbe00640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8604153d70 0x7f8604151550 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f86041f0e20 tx=0x7f85ec00f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:45:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.436+0000 7f85faa00640 1 -- 172.21.15.28:0/1260293241 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ec004020 con 0x7f8604153d70 2024-09-13T16:45:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.436+0000 7f85faa00640 1 -- 172.21.15.28:0/1260293241 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4778 (secure 0 0 0) 0x7f85ec004180 con 0x7f8604153d70 2024-09-13T16:45:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.436+0000 7f85faa00640 1 -- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8604153d70 msgr2=0x7f8604151550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.436+0000 7f85faa00640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8604153d70 0x7f8604151550 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f86041f0e20 tx=0x7f85ec00f610 comp rx=0 tx=0).stop 2024-09-13T16:45:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.436+0000 7f85faa00640 1 -- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8604153d70 msgr2=0x7f8604151550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:45:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.436+0000 7f85faa00640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8604153d70 0x7f8604151550 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f86041f0e20 tx=0x7f85ec00f610 comp rx=0 tx=0).stop 2024-09-13T16:45:50.438 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:45:50.439 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:45:50.439 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:45:50.439 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:45:50.439 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 14, 2024-09-13T16:45:50.439 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:45:50.439 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:45:50.439 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:45:50.439 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:50.439 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 0, 2024-09-13T16:45:50.439 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:45:50.439 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:45:50.439 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:45:50.440 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:50.440 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:50.440 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:50.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:50.440 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:50.440 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:50.440 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:50.440 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:50.440 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:50.440 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:50.440 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:50.440 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:45:50.441 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:45:50.441 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:50.441 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:50.441 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:50.441 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:50.444 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:50.444 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:50.444 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:50.444 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:50.444 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:50.445 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:50.445 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:50.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.445 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:45:50.445 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:45:50.445 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:45:50.445 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:45:50.445 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:45:50.445 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:45:50.445 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:45:50.445 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:45:50.445 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:45:50.445 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:45:50.445 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:45:50.446 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:45:50.446 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:45:50.446 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:45:50.446 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:45:50.446 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:45:50.446 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:45:50.446 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:45:50.446 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:45:50.446 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:45:50.446 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:45:50.446 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:45:50.446 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:45:50.446 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:45:50.447 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:45:50.447 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:45:50.447 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:45:50.447 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:50.447 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:45:50.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.447 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:45:50.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.447 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:45:50.447 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:45:50.447 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:50.447 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:50.447 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:50.448 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:45:50.448 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:50.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.448 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:50.448 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:45:50.448 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:50.448 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:50.448 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:50.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.448 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:45:50.448 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:45:50.449 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:50.449 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:50.449 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:50.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.449 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:45:50.449 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:45:50.449 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:50.449 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:50.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.449 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:50.449 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:45:50.449 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:50.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.450 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:50.450 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:45:50.450 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:50.450 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:50.450 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:50.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.450 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:45:50.450 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:45:50.450 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:50.450 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:50.450 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:50.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.451 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:45:50.451 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:45:50.451 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:45:50.451 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:45:50.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.451 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:45:50.451 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:45:50.451 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:50.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.451 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.451 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:45:50.451 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:45:50.451 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:45:50.452 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:50.452 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:50.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.452 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:45:50.452 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:45:50.452 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:45:50.453 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:45:50.453 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:45:50.453 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:50.453 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:50.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.454 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:45:50.454 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:45:50.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.454 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:50.454 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:50.454 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:45:50.454 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:50.454 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:50.454 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:45:50.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.454 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:50.454 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:50.454 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:45:50.454 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:50.455 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:50.455 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:45:50.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.455 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:50.455 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:50.455 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:45:50.455 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:50.455 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:45:50.455 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:45:50.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:45:50.455 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:45:50.455 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:45:50.455 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:45:50.455 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:45:50.456 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:45:50.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:45:50.456 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:45:50.456 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:45:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.437+0000 7f8609600640 1 -- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f85e006a940 msgr2=0x7f85e006ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.437+0000 7f8609600640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f85e006a940 0x7f85e006ce00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f860406ba20 tx=0x7f85f403d000 comp rx=0 tx=0).stop 2024-09-13T16:45:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.437+0000 7f8609600640 1 -- 172.21.15.28:0/1260293241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f860415a440 msgr2=0x7f86041e0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:45:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.437+0000 7f8609600640 1 --2- 172.21.15.28:0/1260293241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f860415a440 0x7f86041e0100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00b9c0 tx=0x7f85fc00be80 comp rx=0 tx=0).stop 2024-09-13T16:45:50.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.437+0000 7f8603400640 1 -- 172.21.15.28:0/1260293241 reap_dead start 2024-09-13T16:45:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.437+0000 7f8603400640 1 -- 172.21.15.28:0/1260293241 reap_dead start 2024-09-13T16:45:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.437+0000 7f8609600640 1 -- 172.21.15.28:0/1260293241 shutdown_connections 2024-09-13T16:45:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.437+0000 7f8609600640 1 -- 172.21.15.28:0/1260293241 >> 172.21.15.28:0/1260293241 conn(0x7f86040ca6b0 msgr2=0x7f8604069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:45:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.437+0000 7f8609600640 1 -- 172.21.15.28:0/1260293241 shutdown_connections 2024-09-13T16:45:50.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:45:50.437+0000 7f8609600640 1 -- 172.21.15.28:0/1260293241 wait complete. 2024-09-13T16:45:50.458 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-13T16:45:50.458 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-13T16:45:51.218 INFO:tasks.workunit.client.0.smithi028.stderr:Note: switching to '50bbe7f395997dcd871813949c49addbbf630361'. 2024-09-13T16:45:51.218 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-13T16:45:51.218 INFO:tasks.workunit.client.0.smithi028.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-13T16:45:51.218 INFO:tasks.workunit.client.0.smithi028.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-13T16:45:51.218 INFO:tasks.workunit.client.0.smithi028.stderr:state without impacting any branches by switching back to a branch. 2024-09-13T16:45:51.218 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-13T16:45:51.218 INFO:tasks.workunit.client.0.smithi028.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-13T16:45:51.218 INFO:tasks.workunit.client.0.smithi028.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-13T16:45:51.218 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-13T16:45:51.219 INFO:tasks.workunit.client.0.smithi028.stderr: git switch -c 2024-09-13T16:45:51.219 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-13T16:45:51.219 INFO:tasks.workunit.client.0.smithi028.stderr:Or undo this operation with: 2024-09-13T16:45:51.219 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-13T16:45:51.219 INFO:tasks.workunit.client.0.smithi028.stderr: git switch - 2024-09-13T16:45:51.219 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-13T16:45:51.219 INFO:tasks.workunit.client.0.smithi028.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-13T16:45:51.219 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-09-13T16:45:51.219 INFO:tasks.workunit.client.0.smithi028.stderr:HEAD is now at 50bbe7f3959 Merge PR #58789 into wip-rishabh-testing-20240910.095512-reef 2024-09-13T16:45:51.236 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-13T16:45:51.290 INFO:tasks.workunit.client.0.smithi028.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-13T16:45:51.294 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T16:45:51.295 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-13T16:45:51.949 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-13T16:45:52.047 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-13T16:45:52.113 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T16:45:52.116 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T16:45:52.116 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-13T16:45:52.182 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T16:45:52.193 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:45:52.193 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-13T16:45:52.228 INFO:tasks.workunit:Running workunits matching suites/iozone.sh on client.0... 2024-09-13T16:45:52.229 INFO:tasks.workunit:Running workunit suites/iozone.sh... 2024-09-13T16:45:52.230 DEBUG:teuthology.orchestra.run.smithi028:workunit test suites/iozone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bbe7f395997dcd871813949c49addbbf630361 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh 2024-09-13T16:45:52.319 INFO:tasks.workunit.client.0.smithi028.stderr:+ iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-09-13T16:45:52.422 INFO:tasks.workunit.client.0.smithi028.stdout: Iozone: Performance Test of File I/O 2024-09-13T16:45:52.422 INFO:tasks.workunit.client.0.smithi028.stdout: Version $Revision: 3.494 $ 2024-09-13T16:45:52.423 INFO:tasks.workunit.client.0.smithi028.stdout: Compiled for 64 bit mode. 2024-09-13T16:45:52.423 INFO:tasks.workunit.client.0.smithi028.stdout: Build: linux-AMD64 2024-09-13T16:45:52.423 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:45:52.423 INFO:tasks.workunit.client.0.smithi028.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-09-13T16:45:52.423 INFO:tasks.workunit.client.0.smithi028.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-09-13T16:45:52.423 INFO:tasks.workunit.client.0.smithi028.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-09-13T16:45:52.423 INFO:tasks.workunit.client.0.smithi028.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-09-13T16:45:52.423 INFO:tasks.workunit.client.0.smithi028.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-09-13T16:45:52.423 INFO:tasks.workunit.client.0.smithi028.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-09-13T16:45:52.423 INFO:tasks.workunit.client.0.smithi028.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-09-13T16:45:52.423 INFO:tasks.workunit.client.0.smithi028.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-09-13T16:45:52.423 INFO:tasks.workunit.client.0.smithi028.stdout: Alexey Skidanov, Sudhir Kumar. 2024-09-13T16:45:52.423 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:45:52.424 INFO:tasks.workunit.client.0.smithi028.stdout: Run began: Fri Sep 13 16:45:52 2024 2024-09-13T16:45:52.424 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:45:52.424 INFO:tasks.workunit.client.0.smithi028.stdout: Include close in write timing 2024-09-13T16:45:52.424 INFO:tasks.workunit.client.0.smithi028.stdout: Include fsync in write timing 2024-09-13T16:45:52.424 INFO:tasks.workunit.client.0.smithi028.stdout: File size set to 1048576 kB 2024-09-13T16:45:52.424 INFO:tasks.workunit.client.0.smithi028.stdout: Record Size 16 kB 2024-09-13T16:45:52.424 INFO:tasks.workunit.client.0.smithi028.stdout: Command line used: iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-09-13T16:45:52.424 INFO:tasks.workunit.client.0.smithi028.stdout: Output is in kBytes/sec 2024-09-13T16:45:52.424 INFO:tasks.workunit.client.0.smithi028.stdout: Time Resolution = 0.000001 seconds. 2024-09-13T16:45:52.424 INFO:tasks.workunit.client.0.smithi028.stdout: Processor cache size set to 1024 kBytes. 2024-09-13T16:45:52.424 INFO:tasks.workunit.client.0.smithi028.stdout: Processor cache line size set to 32 bytes. 2024-09-13T16:45:52.424 INFO:tasks.workunit.client.0.smithi028.stdout: File stride size set to 17 * record size. 2024-09-13T16:45:52.424 INFO:tasks.workunit.client.0.smithi028.stdout: Throughput test with 1 process 2024-09-13T16:45:52.425 INFO:tasks.workunit.client.0.smithi028.stdout: Each process writes a 1048576 kByte file in 16 kByte records 2024-09-13T16:45:55.865 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:45:55.865 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 initial writers = 304913.62 kB/sec 2024-09-13T16:45:55.866 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 initial writers = 304856.61 kB/sec 2024-09-13T16:45:55.866 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 304913.62 kB/sec 2024-09-13T16:45:55.866 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 304913.62 kB/sec 2024-09-13T16:45:55.866 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 304913.62 kB/sec 2024-09-13T16:45:55.866 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 1048576.00 kB 2024-09-13T16:46:02.883 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:02.883 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 rewriters = 347406.50 kB/sec 2024-09-13T16:46:02.883 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 rewriters = 347332.87 kB/sec 2024-09-13T16:46:02.883 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 347406.50 kB/sec 2024-09-13T16:46:02.883 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 347406.50 kB/sec 2024-09-13T16:46:02.883 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 347406.50 kB/sec 2024-09-13T16:46:02.884 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 1048576.00 kB 2024-09-13T16:46:03.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:46:03.743 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:46:05.636 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:05.637 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 readers = 3498482.50 kB/sec 2024-09-13T16:46:05.637 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 readers = 3492295.82 kB/sec 2024-09-13T16:46:05.637 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 3498482.50 kB/sec 2024-09-13T16:46:05.637 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 3498482.50 kB/sec 2024-09-13T16:46:05.637 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 3498482.50 kB/sec 2024-09-13T16:46:05.637 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 1048576.00 kB 2024-09-13T16:46:08.150 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:08.150 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 re-readers = 2652714.75 kB/sec 2024-09-13T16:46:08.150 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 re-readers = 2649070.05 kB/sec 2024-09-13T16:46:08.150 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 2652714.75 kB/sec 2024-09-13T16:46:08.151 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 2652714.75 kB/sec 2024-09-13T16:46:08.151 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 2652714.75 kB/sec 2024-09-13T16:46:08.151 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 1048576.00 kB 2024-09-13T16:46:10.459 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-13T16:46:10.460 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-13T16:46:10.460 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-13T16:46:10.460 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-13T16:46:10.463 INFO:tasks.ceph.mon.b:Started 2024-09-13T16:46:10.463 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T16:46:10.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:46:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.628+0000 7fa02cc00640 1 Processor -- start 2024-09-13T16:46:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.628+0000 7fa02cc00640 1 -- start start 2024-09-13T16:46:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa02cc00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa028150a70 0x7fa0280c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa02cc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa0280c6720 0x7fa0280c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa02cc00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa0280c53a0 0x7fa028156c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa02cc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0280c64f0 con 0x7fa0280c6720 2024-09-13T16:46:10.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa02cc00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa028159f70 con 0x7fa0280c53a0 2024-09-13T16:46:10.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa02cc00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0280c1ea0 con 0x7fa028150a70 2024-09-13T16:46:10.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa01fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa0280c6720 0x7fa0280c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:10.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa01fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa0280c6720 0x7fa0280c4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51584/0 (socket says 172.21.15.28:51584) 2024-09-13T16:46:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa01fe00640 1 -- 172.21.15.28:0/2005677387 learned_addr learned my addr 172.21.15.28:0/2005677387 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa025e00640 1 --2- 172.21.15.28:0/2005677387 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa028150a70 0x7fa0280c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa026800640 1 --2- 172.21.15.28:0/2005677387 >> v2:172.21.15.191:3300/0 conn(0x7fa0280c53a0 0x7fa028156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa026800640 1 -- 172.21.15.28:0/2005677387 >> v2:172.21.15.191:3300/0 conn(0x7fa0280c53a0 msgr2=0x7fa028156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T16:46:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa026800640 1 -- 172.21.15.28:0/2005677387 >> v2:172.21.15.191:3300/0 conn(0x7fa0280c53a0 msgr2=0x7fa028156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:46:10.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa026800640 1 --2- 172.21.15.28:0/2005677387 >> v2:172.21.15.191:3300/0 conn(0x7fa0280c53a0 0x7fa028156c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:46:10.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa01fe00640 1 -- 172.21.15.28:0/2005677387 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa028150a70 msgr2=0x7fa0280c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:46:10.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa01fe00640 1 --2- 172.21.15.28:0/2005677387 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa028150a70 0x7fa0280c1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:10.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa01fe00640 1 -- 172.21.15.28:0/2005677387 >> v2:172.21.15.191:3300/0 conn(0x7fa0280c53a0 msgr2=0x7fa028156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:46:10.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa026800640 1 --2- 172.21.15.28:0/2005677387 >> v2:172.21.15.191:3300/0 conn(0x7fa0280c53a0 0x7fa028156c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:10.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa01fe00640 1 --2- 172.21.15.28:0/2005677387 >> v2:172.21.15.191:3300/0 conn(0x7fa0280c53a0 0x7fa028156c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:10.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.630+0000 7fa01fe00640 1 -- 172.21.15.28:0/2005677387 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02815a1e0 con 0x7fa0280c6720 2024-09-13T16:46:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.632+0000 7fa01fe00640 1 --2- 172.21.15.28:0/2005677387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0280c6720 0x7fa0280c4e60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa028072580 tx=0x7fa01002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bbfaf969c4571f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.632+0000 7fa025400640 1 -- 172.21.15.28:0/2005677387 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa010032440 con 0x7fa0280c6720 2024-09-13T16:46:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.632+0000 7fa025400640 1 -- 172.21.15.28:0/2005677387 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0100325a0 con 0x7fa0280c6720 2024-09-13T16:46:10.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.632+0000 7fa025400640 1 -- 172.21.15.28:0/2005677387 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa010032930 con 0x7fa0280c6720 2024-09-13T16:46:10.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.632+0000 7fa02cc00640 1 -- 172.21.15.28:0/2005677387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0280c6720 msgr2=0x7fa0280c4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:10.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.632+0000 7fa02cc00640 1 --2- 172.21.15.28:0/2005677387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0280c6720 0x7fa0280c4e60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa028072580 tx=0x7fa01002a270 comp rx=0 tx=0).stop 2024-09-13T16:46:10.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.633+0000 7fa02cc00640 1 -- 172.21.15.28:0/2005677387 shutdown_connections 2024-09-13T16:46:10.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.633+0000 7fa02cc00640 1 --2- 172.21.15.28:0/2005677387 >> v2:172.21.15.191:3300/0 conn(0x7fa0280c53a0 0x7fa028156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:10.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.633+0000 7fa02cc00640 1 --2- 172.21.15.28:0/2005677387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0280c6720 0x7fa0280c4e60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:10.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.633+0000 7fa02cc00640 1 --2- 172.21.15.28:0/2005677387 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa028150a70 0x7fa0280c1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:10.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.633+0000 7fa02cc00640 1 -- 172.21.15.28:0/2005677387 >> 172.21.15.28:0/2005677387 conn(0x7fa028069e80 msgr2=0x7fa02806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:10.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.633+0000 7fa02cc00640 1 -- 172.21.15.28:0/2005677387 shutdown_connections 2024-09-13T16:46:10.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.633+0000 7fa02cc00640 1 -- 172.21.15.28:0/2005677387 wait complete. 2024-09-13T16:46:10.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.634+0000 7fa02cc00640 1 Processor -- start 2024-09-13T16:46:10.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.634+0000 7fa02cc00640 1 -- start start 2024-09-13T16:46:10.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.635+0000 7fa02cc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0280c53a0 0x7fa028159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.635+0000 7fa02cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0280c6720 0x7fa028152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.635+0000 7fa02cc00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa028150a70 0x7fa0281ec550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.635+0000 7fa02cc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0281ef850 con 0x7fa0280c6720 2024-09-13T16:46:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.635+0000 7fa02cc00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0281f06b0 con 0x7fa0280c53a0 2024-09-13T16:46:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.635+0000 7fa02cc00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0281efaa0 con 0x7fa028150a70 2024-09-13T16:46:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.636+0000 7fa025e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0280c53a0 0x7fa028159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.636+0000 7fa026800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa028150a70 0x7fa0281ec550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:10.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.636+0000 7fa025e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0280c53a0 0x7fa028159630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60058/0 (socket says 172.21.15.28:60058) 2024-09-13T16:46:10.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.636+0000 7fa025e00640 1 -- 172.21.15.28:0/2054307311 learned_addr learned my addr 172.21.15.28:0/2054307311 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:10.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.636+0000 7fa01fe00640 1 --2- 172.21.15.28:0/2054307311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0280c6720 0x7fa028152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:10.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.636+0000 7fa025e00640 1 -- 172.21.15.28:0/2054307311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0280c53a0 msgr2=0x7fa028159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-13T16:46:10.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.636+0000 7fa025e00640 1 -- 172.21.15.28:0/2054307311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0280c53a0 msgr2=0x7fa028159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T16:46:10.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.636+0000 7fa025e00640 1 --2- 172.21.15.28:0/2054307311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0280c53a0 0x7fa028159630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:46:10.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.636+0000 7fa025e00640 1 --2- 172.21.15.28:0/2054307311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0280c53a0 0x7fa028159630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:10.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.636+0000 7fa026800640 1 -- 172.21.15.28:0/2054307311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0280c53a0 msgr2=0x7fa028159630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:46:10.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.636+0000 7fa026800640 1 --2- 172.21.15.28:0/2054307311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0280c53a0 0x7fa028159630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:10.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.636+0000 7fa026800640 1 -- 172.21.15.28:0/2054307311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0280c6720 msgr2=0x7fa028152660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:10.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.636+0000 7fa026800640 1 --2- 172.21.15.28:0/2054307311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0280c6720 0x7fa028152660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:10.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.636+0000 7fa026800640 1 -- 172.21.15.28:0/2054307311 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0100094d0 con 0x7fa028150a70 2024-09-13T16:46:10.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.636+0000 7fa026800640 1 --2- 172.21.15.28:0/2054307311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa028150a70 0x7fa0281ec550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa020010e40 tx=0x7fa020007700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:10.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.637+0000 7fa01e000640 1 -- 172.21.15.28:0/2054307311 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa020015070 con 0x7fa028150a70 2024-09-13T16:46:10.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.637+0000 7fa01e000640 1 -- 172.21.15.28:0/2054307311 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa020007e70 con 0x7fa028150a70 2024-09-13T16:46:10.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.637+0000 7fa01e000640 1 -- 172.21.15.28:0/2054307311 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02000e670 con 0x7fa028150a70 2024-09-13T16:46:10.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.637+0000 7fa02cc00640 1 -- 172.21.15.28:0/2054307311 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0281efd80 con 0x7fa028150a70 2024-09-13T16:46:10.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.637+0000 7fa02cc00640 1 -- 172.21.15.28:0/2054307311 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0281f0340 con 0x7fa028150a70 2024-09-13T16:46:10.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.638+0000 7fa02cc00640 1 -- 172.21.15.28:0/2054307311 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0280c4750 con 0x7fa028150a70 2024-09-13T16:46:10.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.639+0000 7fa01e000640 1 -- 172.21.15.28:0/2054307311 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa02000e7d0 con 0x7fa028150a70 2024-09-13T16:46:10.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.640+0000 7fa01e000640 1 --2- 172.21.15.28:0/2054307311 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa00406ac30 0x7fa00406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:10.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.640+0000 7fa01e000640 1 -- 172.21.15.28:0/2054307311 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa02008d250 con 0x7fa028150a70 2024-09-13T16:46:10.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.640+0000 7fa025e00640 1 --2- 172.21.15.28:0/2054307311 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa00406ac30 0x7fa00406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:10.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.640+0000 7fa025e00640 1 --2- 172.21.15.28:0/2054307311 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa00406ac30 0x7fa00406d0f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa018009440 tx=0x7fa018008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:10.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.644+0000 7fa01e000640 1 -- 172.21.15.28:0/2054307311 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa0200583c0 con 0x7fa028150a70 2024-09-13T16:46:10.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.890+0000 7fa02cc00640 1 -- 172.21.15.28:0/2054307311 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa0280c5930 con 0x7fa028150a70 2024-09-13T16:46:10.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.890+0000 7fa01e000640 1 -- 172.21.15.28:0/2054307311 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7fa02005b9e0 con 0x7fa028150a70 2024-09-13T16:46:10.892 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:46:10.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.893+0000 7fa02cc00640 1 -- 172.21.15.28:0/2054307311 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa00406ac30 msgr2=0x7fa00406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:10.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.893+0000 7fa02cc00640 1 --2- 172.21.15.28:0/2054307311 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa00406ac30 0x7fa00406d0f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa018009440 tx=0x7fa018008040 comp rx=0 tx=0).stop 2024-09-13T16:46:10.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.893+0000 7fa02cc00640 1 -- 172.21.15.28:0/2054307311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa028150a70 msgr2=0x7fa0281ec550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:10.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.893+0000 7fa02cc00640 1 --2- 172.21.15.28:0/2054307311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa028150a70 0x7fa0281ec550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa020010e40 tx=0x7fa020007700 comp rx=0 tx=0).stop 2024-09-13T16:46:10.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.893+0000 7fa02cc00640 1 -- 172.21.15.28:0/2054307311 shutdown_connections 2024-09-13T16:46:10.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.893+0000 7fa02cc00640 1 --2- 172.21.15.28:0/2054307311 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa00406ac30 0x7fa00406d0f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:10.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.893+0000 7fa02cc00640 1 --2- 172.21.15.28:0/2054307311 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa028150a70 0x7fa0281ec550 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:10.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.893+0000 7fa02cc00640 1 --2- 172.21.15.28:0/2054307311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0280c6720 0x7fa028152660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:10.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.893+0000 7fa02cc00640 1 --2- 172.21.15.28:0/2054307311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa0280c53a0 0x7fa028159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:10.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.893+0000 7fa02cc00640 1 -- 172.21.15.28:0/2054307311 >> 172.21.15.28:0/2054307311 conn(0x7fa028069e80 msgr2=0x7fa02814f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:10.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.893+0000 7fa02cc00640 1 -- 172.21.15.28:0/2054307311 shutdown_connections 2024-09-13T16:46:10.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:10.893+0000 7fa02cc00640 1 -- 172.21.15.28:0/2054307311 wait complete. 2024-09-13T16:46:10.911 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T16:46:10.911 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-13T16:46:10.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T16:46:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.068+0000 7ff921600640 1 Processor -- start 2024-09-13T16:46:11.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.068+0000 7ff921600640 1 -- start start 2024-09-13T16:46:11.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.069+0000 7ff921600640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff91c15a060 0x7ff91c0c5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.069+0000 7ff921600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff91c06dcb0 0x7ff91c1504e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.069+0000 7ff921600640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff91c0c1d80 con 0x7ff91c15a440 2024-09-13T16:46:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.069+0000 7ff921600640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff91c15a920 con 0x7ff91c15a060 2024-09-13T16:46:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.069+0000 7ff921600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c15ac00 con 0x7ff91c06dcb0 2024-09-13T16:46:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.070+0000 7ff91b400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff91c06dcb0 0x7ff91c1504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.070+0000 7ff91aa00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff91c15a060 0x7ff91c0c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.070+0000 7ff91a000640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7ff91c15a440 0x7ff91c06d5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:32918/0 (socket says 172.21.15.28:32918) 2024-09-13T16:46:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.070+0000 7ff91aa00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff91c15a060 0x7ff91c0c5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:60060/0 (socket says 172.21.15.28:60060) 2024-09-13T16:46:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.070+0000 7ff91aa00640 1 -- 172.21.15.28:0/1412251420 learned_addr learned my addr 172.21.15.28:0/1412251420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.070+0000 7ff91b400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff91c06dcb0 0x7ff91c1504e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:60938/0 (socket says 172.21.15.28:60938) 2024-09-13T16:46:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.070+0000 7ff91aa00640 1 -- 172.21.15.28:0/1412251420 >> v2:172.21.15.191:3300/0 conn(0x7ff91c15a060 msgr2=0x7ff91c0c5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:46:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.070+0000 7ff91aa00640 1 -- 172.21.15.28:0/1412251420 >> v2:172.21.15.191:3300/0 conn(0x7ff91c15a060 msgr2=0x7ff91c0c5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:46:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.070+0000 7ff91aa00640 1 --2- 172.21.15.28:0/1412251420 >> v2:172.21.15.191:3300/0 conn(0x7ff91c15a060 0x7ff91c0c5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:46:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.070+0000 7ff91aa00640 1 --2- 172.21.15.28:0/1412251420 >> v2:172.21.15.191:3300/0 conn(0x7ff91c15a060 0x7ff91c0c5500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.070+0000 7ff91b400640 1 -- 172.21.15.28:0/1412251420 >> v2:172.21.15.191:3300/0 conn(0x7ff91c15a060 msgr2=0x7ff91c0c5500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:46:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.070+0000 7ff91b400640 1 --2- 172.21.15.28:0/1412251420 >> v2:172.21.15.191:3300/0 conn(0x7ff91c15a060 0x7ff91c0c5500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.070+0000 7ff91b400640 1 -- 172.21.15.28:0/1412251420 >> v1:172.21.15.191:6789/0 conn(0x7ff91c15a440 legacy=0x7ff91c06d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.070+0000 7ff91b400640 1 -- 172.21.15.28:0/1412251420 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c0c2d50 con 0x7ff91c06dcb0 2024-09-13T16:46:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.071+0000 7ff91b400640 1 --2- 172.21.15.28:0/1412251420 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff91c06dcb0 0x7ff91c1504e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff914009d90 tx=0x7ff91402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0476b31e0820267 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.071+0000 7ff919600640 1 -- 172.21.15.28:0/1412251420 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff914007740 con 0x7ff91c06dcb0 2024-09-13T16:46:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.071+0000 7ff919600640 1 -- 172.21.15.28:0/1412251420 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91402d050 con 0x7ff91c06dcb0 2024-09-13T16:46:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.071+0000 7ff919600640 1 -- 172.21.15.28:0/1412251420 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff914007ac0 con 0x7ff91c06dcb0 2024-09-13T16:46:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.072+0000 7ff921600640 1 -- 172.21.15.28:0/1412251420 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff91c06dcb0 msgr2=0x7ff91c1504e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.072+0000 7ff921600640 1 --2- 172.21.15.28:0/1412251420 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff91c06dcb0 0x7ff91c1504e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff914009d90 tx=0x7ff91402a660 comp rx=0 tx=0).stop 2024-09-13T16:46:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.072+0000 7ff921600640 1 -- 172.21.15.28:0/1412251420 shutdown_connections 2024-09-13T16:46:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.072+0000 7ff921600640 1 --2- 172.21.15.28:0/1412251420 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff91c06dcb0 0x7ff91c1504e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.072+0000 7ff921600640 1 --2- 172.21.15.28:0/1412251420 >> v2:172.21.15.191:3300/0 conn(0x7ff91c15a060 0x7ff91c0c5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.072+0000 7ff921600640 1 -- 172.21.15.28:0/1412251420 >> 172.21.15.28:0/1412251420 conn(0x7ff91c0ca6b0 msgr2=0x7ff91c0caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:11.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.072+0000 7ff921600640 1 -- 172.21.15.28:0/1412251420 shutdown_connections 2024-09-13T16:46:11.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.072+0000 7ff921600640 1 -- 172.21.15.28:0/1412251420 wait complete. 2024-09-13T16:46:11.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.074+0000 7ff921600640 1 Processor -- start 2024-09-13T16:46:11.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.074+0000 7ff921600640 1 -- start start 2024-09-13T16:46:11.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.075+0000 7ff921600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff91c06dcb0 0x7ff91c14c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.075+0000 7ff921600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c15a060 0x7ff91c1f2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.075+0000 7ff921600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff91c15a440 0x7ff91c148730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.075+0000 7ff921600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c14ccc0 con 0x7ff91c15a060 2024-09-13T16:46:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.075+0000 7ff921600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c14d4b0 con 0x7ff91c15a440 2024-09-13T16:46:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.075+0000 7ff921600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c14b4c0 con 0x7ff91c06dcb0 2024-09-13T16:46:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.075+0000 7ff91a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c15a060 0x7ff91c1f2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.075+0000 7ff91aa00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff91c06dcb0 0x7ff91c14c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.075+0000 7ff91aa00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff91c06dcb0 0x7ff91c14c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:60944/0 (socket says 172.21.15.28:60944) 2024-09-13T16:46:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.075+0000 7ff91a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c15a060 0x7ff91c1f2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51590/0 (socket says 172.21.15.28:51590) 2024-09-13T16:46:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.075+0000 7ff91aa00640 1 -- 172.21.15.28:0/3660114213 learned_addr learned my addr 172.21.15.28:0/3660114213 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.075+0000 7ff91b400640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff91c15a440 0x7ff91c148730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.075+0000 7ff91aa00640 1 -- 172.21.15.28:0/3660114213 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff91c15a440 msgr2=0x7ff91c148730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.075+0000 7ff91aa00640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff91c15a440 0x7ff91c148730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.075+0000 7ff91aa00640 1 -- 172.21.15.28:0/3660114213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c15a060 msgr2=0x7ff91c1f2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.075+0000 7ff91aa00640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c15a060 0x7ff91c1f2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.076+0000 7ff91aa00640 1 -- 172.21.15.28:0/3660114213 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff914009a70 con 0x7ff91c06dcb0 2024-09-13T16:46:11.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.076+0000 7ff91b400640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff91c15a440 0x7ff91c148730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:46:11.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.076+0000 7ff91a000640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c15a060 0x7ff91c1f2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:46:11.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.076+0000 7ff91aa00640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff91c06dcb0 0x7ff91c14c780 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff90c00e810 tx=0x7ff90c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.076+0000 7ff912a00640 1 -- 172.21.15.28:0/3660114213 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90c004020 con 0x7ff91c06dcb0 2024-09-13T16:46:11.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.076+0000 7ff912a00640 1 -- 172.21.15.28:0/3660114213 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff90c004180 con 0x7ff91c06dcb0 2024-09-13T16:46:11.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.076+0000 7ff912a00640 1 -- 172.21.15.28:0/3660114213 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90c00c670 con 0x7ff91c06dcb0 2024-09-13T16:46:11.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.076+0000 7ff921600640 1 -- 172.21.15.28:0/3660114213 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff91c14af80 con 0x7ff91c06dcb0 2024-09-13T16:46:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.076+0000 7ff921600640 1 -- 172.21.15.28:0/3660114213 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff91c14aca0 con 0x7ff91c06dcb0 2024-09-13T16:46:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.077+0000 7ff921600640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c14d080 0x7ff91c148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.077+0000 7ff921600640 1 -- 172.21.15.28:0/3660114213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c066c20 con 0x7ff91c14d080 2024-09-13T16:46:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.077+0000 7ff91a000640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c14d080 0x7ff91c148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.078+0000 7ff91a000640 1 -- 172.21.15.28:0/3660114213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff91c0c5220 con 0x7ff91c14d080 2024-09-13T16:46:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.078+0000 7ff91a000640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c14d080 0x7ff91c148c70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff91c14a820 tx=0x7ff908001c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.078+0000 7ff912a00640 1 -- 172.21.15.28:0/3660114213 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90800f9b0 con 0x7ff91c14d080 2024-09-13T16:46:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.078+0000 7ff912a00640 1 -- 172.21.15.28:0/3660114213 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff90c00c7d0 con 0x7ff91c06dcb0 2024-09-13T16:46:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.079+0000 7ff912a00640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff8fc06aca0 0x7ff8fc06d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.079+0000 7ff912a00640 1 -- 172.21.15.28:0/3660114213 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff90c088f00 con 0x7ff91c06dcb0 2024-09-13T16:46:11.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.079+0000 7ff912a00640 1 -- 172.21.15.28:0/3660114213 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7ff90800fb10 con 0x7ff91c14d080 2024-09-13T16:46:11.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.079+0000 7ff912a00640 1 -- 172.21.15.28:0/3660114213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c14d080 msgr2=0x7ff91c148c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.079+0000 7ff912a00640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c14d080 0x7ff91c148c70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff91c14a820 tx=0x7ff908001c60 comp rx=0 tx=0).stop 2024-09-13T16:46:11.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.079+0000 7ff912a00640 1 -- 172.21.15.28:0/3660114213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c14d080 msgr2=0x7ff91c148c70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:46:11.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.079+0000 7ff912a00640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c14d080 0x7ff91c148c70 secure :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff91c14a820 tx=0x7ff908001c60 comp rx=0 tx=0).stop 2024-09-13T16:46:11.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.080+0000 7ff91b400640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff8fc06aca0 0x7ff8fc06d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.080+0000 7ff91b400640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff8fc06aca0 0x7ff8fc06d160 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff914034000 tx=0x7ff91402f000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.086+0000 7ff921600640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c14a220 0x7ff91c158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.086+0000 7ff921600640 1 -- 172.21.15.28:0/3660114213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c0c3310 con 0x7ff91c14a220 2024-09-13T16:46:11.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.087+0000 7ff91a000640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c14a220 0x7ff91c158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.087+0000 7ff91a000640 1 -- 172.21.15.28:0/3660114213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff91c0c19d0 con 0x7ff91c14a220 2024-09-13T16:46:11.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.087+0000 7ff91a000640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c14a220 0x7ff91c158a80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff91c144e90 tx=0x7ff908007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.087+0000 7ff912a00640 1 -- 172.21.15.28:0/3660114213 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff908022070 con 0x7ff91c14a220 2024-09-13T16:46:11.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.088+0000 7ff912a00640 1 -- 172.21.15.28:0/3660114213 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+5100 (secure 0 0 0) 0x7ff9080136c0 con 0x7ff91c14a220 2024-09-13T16:46:11.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.088+0000 7ff912a00640 1 -- 172.21.15.28:0/3660114213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c14a220 msgr2=0x7ff91c158a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.088+0000 7ff912a00640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c14a220 0x7ff91c158a80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff91c144e90 tx=0x7ff908007f90 comp rx=0 tx=0).stop 2024-09-13T16:46:11.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.088+0000 7ff912a00640 1 -- 172.21.15.28:0/3660114213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c14a220 msgr2=0x7ff91c158a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:46:11.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.088+0000 7ff912a00640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff91c14a220 0x7ff91c158a80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "leader", 2024-09-13T16:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 16, 2024-09-13T16:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:46:11.090 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 0, 2024-09-13T16:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:46:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:46:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:46:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:46:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:46:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:46:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:46:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:46:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:46:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:46:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:46:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:46:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:46:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.096 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.097 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:11.098 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.100 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:11.101 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 5 2024-09-13T16:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x2f018fb87aa4aafe", 2024-09-13T16:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "luminous", 2024-09-13T16:46:11.102 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:46:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4 2024-09-13T16:46:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": [ 2024-09-13T16:46:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:46:11.103 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.104 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:46:11.104 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:46:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.088+0000 7ff921600640 1 -- 172.21.15.28:0/3660114213 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff8fc06aca0 msgr2=0x7ff8fc06d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.088+0000 7ff921600640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff8fc06aca0 0x7ff8fc06d160 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff914034000 tx=0x7ff91402f000 comp rx=0 tx=0).stop 2024-09-13T16:46:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.088+0000 7ff921600640 1 -- 172.21.15.28:0/3660114213 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff91c06dcb0 msgr2=0x7ff91c14c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.088+0000 7ff921600640 1 --2- 172.21.15.28:0/3660114213 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff91c06dcb0 0x7ff91c14c780 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff90c00e810 tx=0x7ff90c00ecd0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.088+0000 7ff91b400640 1 -- 172.21.15.28:0/3660114213 reap_dead start 2024-09-13T16:46:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.088+0000 7ff91b400640 1 -- 172.21.15.28:0/3660114213 reap_dead start 2024-09-13T16:46:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.088+0000 7ff921600640 1 -- 172.21.15.28:0/3660114213 shutdown_connections 2024-09-13T16:46:11.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.088+0000 7ff921600640 1 -- 172.21.15.28:0/3660114213 >> 172.21.15.28:0/3660114213 conn(0x7ff91c0ca6b0 msgr2=0x7ff91c14fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:11.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.088+0000 7ff921600640 1 -- 172.21.15.28:0/3660114213 shutdown_connections 2024-09-13T16:46:11.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.088+0000 7ff921600640 1 -- 172.21.15.28:0/3660114213 wait complete. 2024-09-13T16:46:11.106 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:46:11.106 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-13T16:46:11.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T16:46:11.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.261+0000 7f97d2c00640 1 Processor -- start 2024-09-13T16:46:11.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.261+0000 7f97d2c00640 1 -- start start 2024-09-13T16:46:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.262+0000 7f97d2c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97cc15a060 0x7f97cc0c5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.262+0000 7f97d2c00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f97cc06dcb0 0x7f97cc1504e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.262+0000 7f97d2c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97cc0c1d80 con 0x7f97cc15a440 2024-09-13T16:46:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.262+0000 7f97d2c00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97cc15a920 con 0x7f97cc06dcb0 2024-09-13T16:46:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.262+0000 7f97d2c00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc15ac00 con 0x7f97cc15a060 2024-09-13T16:46:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.262+0000 7f97c3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f97cc15a440 0x7f97cc06d5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57072/0 (socket says 172.21.15.28:57072) 2024-09-13T16:46:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.262+0000 7f97cbe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97cc15a060 0x7f97cc0c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.262+0000 7f97c3400640 1 -- 172.21.15.28:0/3766794895 learned_addr learned my addr 172.21.15.28:0/3766794895 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:46:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.262+0000 7f97d0c00640 1 --2- 172.21.15.28:0/3766794895 >> v2:172.21.15.191:3300/0 conn(0x7f97cc06dcb0 0x7f97cc1504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.263+0000 7f97cbe00640 1 -- 172.21.15.28:0/3766794895 >> v2:172.21.15.191:3300/0 conn(0x7f97cc06dcb0 msgr2=0x7f97cc1504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:46:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.263+0000 7f97cbe00640 1 --2- 172.21.15.28:0/3766794895 >> v2:172.21.15.191:3300/0 conn(0x7f97cc06dcb0 0x7f97cc1504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.263+0000 7f97cbe00640 1 -- 172.21.15.28:0/3766794895 >> v1:172.21.15.28:6789/0 conn(0x7f97cc15a440 legacy=0x7f97cc06d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.263+0000 7f97cb400640 1 -- 172.21.15.28:0/3766794895 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218663896 0 0) 0x7f97cc0c1d80 con 0x7f97cc15a440 2024-09-13T16:46:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.263+0000 7f97cbe00640 1 -- 172.21.15.28:0/3766794895 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97cc0c2d50 con 0x7f97cc15a060 2024-09-13T16:46:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.264+0000 7f97cbe00640 1 --2- 172.21.15.28:0/3766794895 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97cc15a060 0x7f97cc0c5500 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0097a0 tx=0x7f97bc02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eda0e6cf86a2ac5d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.264+0000 7f97cb400640 1 -- 172.21.15.28:0/3766794895 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc032440 con 0x7f97cc15a060 2024-09-13T16:46:11.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.264+0000 7f97cb400640 1 -- 172.21.15.28:0/3766794895 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97bc0325a0 con 0x7f97cc15a060 2024-09-13T16:46:11.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.264+0000 7f97cb400640 1 -- 172.21.15.28:0/3766794895 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc032930 con 0x7f97cc15a060 2024-09-13T16:46:11.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.265+0000 7f97d2c00640 1 -- 172.21.15.28:0/3766794895 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97cc15a060 msgr2=0x7f97cc0c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.265+0000 7f97d2c00640 1 --2- 172.21.15.28:0/3766794895 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97cc15a060 0x7f97cc0c5500 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0097a0 tx=0x7f97bc02a180 comp rx=0 tx=0).stop 2024-09-13T16:46:11.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.265+0000 7f97d2c00640 1 -- 172.21.15.28:0/3766794895 shutdown_connections 2024-09-13T16:46:11.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.265+0000 7f97d2c00640 1 --2- 172.21.15.28:0/3766794895 >> v2:172.21.15.191:3300/0 conn(0x7f97cc06dcb0 0x7f97cc1504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.265+0000 7f97d2c00640 1 --2- 172.21.15.28:0/3766794895 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97cc15a060 0x7f97cc0c5500 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.265+0000 7f97d2c00640 1 -- 172.21.15.28:0/3766794895 >> 172.21.15.28:0/3766794895 conn(0x7f97cc0ca6b0 msgr2=0x7f97cc0caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:11.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.265+0000 7f97d2c00640 1 -- 172.21.15.28:0/3766794895 shutdown_connections 2024-09-13T16:46:11.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.265+0000 7f97d2c00640 1 -- 172.21.15.28:0/3766794895 wait complete. 2024-09-13T16:46:11.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.266+0000 7f97d2c00640 1 Processor -- start 2024-09-13T16:46:11.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.267+0000 7f97d2c00640 1 -- start start 2024-09-13T16:46:11.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.267+0000 7f97d2c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97cc06dcb0 0x7f97cc1585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.267+0000 7f97d2c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97cc15a060 0x7f97cc1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.267+0000 7f97d2c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc15a440 0x7f97cc148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.267+0000 7f97d2c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc1e1e10 con 0x7f97cc15a060 2024-09-13T16:46:11.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.267+0000 7f97d2c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc1e18e0 con 0x7f97cc15a440 2024-09-13T16:46:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.268+0000 7f97d2c00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc0c7a10 con 0x7f97cc06dcb0 2024-09-13T16:46:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.268+0000 7f97c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97cc15a060 0x7f97cc1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.268+0000 7f97cbe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97cc06dcb0 0x7f97cc1585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.268+0000 7f97c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97cc15a060 0x7f97cc1593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51614/0 (socket says 172.21.15.28:51614) 2024-09-13T16:46:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.268+0000 7f97c3400640 1 -- 172.21.15.28:0/2096680807 learned_addr learned my addr 172.21.15.28:0/2096680807 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.268+0000 7f97d0c00640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc15a440 0x7f97cc148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.268+0000 7f97cbe00640 1 -- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc15a440 msgr2=0x7f97cc148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.268+0000 7f97cbe00640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc15a440 0x7f97cc148eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.268+0000 7f97cbe00640 1 -- 172.21.15.28:0/2096680807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97cc15a060 msgr2=0x7f97cc1593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.268+0000 7f97cbe00640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97cc15a060 0x7f97cc1593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.268+0000 7f97cbe00640 1 -- 172.21.15.28:0/2096680807 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc009400 con 0x7f97cc06dcb0 2024-09-13T16:46:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.268+0000 7f97c3400640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97cc15a060 0x7f97cc1593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:46:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.268+0000 7f97cbe00640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97cc06dcb0 0x7f97cc1585e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f97bc031000 tx=0x7f97bc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.269+0000 7f97c8c00640 1 -- 172.21.15.28:0/2096680807 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc030b80 con 0x7f97cc06dcb0 2024-09-13T16:46:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.269+0000 7f97c8c00640 1 -- 172.21.15.28:0/2096680807 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97bc030ce0 con 0x7f97cc06dcb0 2024-09-13T16:46:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.269+0000 7f97c8c00640 1 -- 172.21.15.28:0/2096680807 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc03d680 con 0x7f97cc06dcb0 2024-09-13T16:46:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.269+0000 7f97d2c00640 1 -- 172.21.15.28:0/2096680807 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97cc1528a0 con 0x7f97cc06dcb0 2024-09-13T16:46:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.269+0000 7f97d2c00640 1 -- 172.21.15.28:0/2096680807 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97cc151430 con 0x7f97cc06dcb0 2024-09-13T16:46:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.270+0000 7f97d2c00640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc0c7560 0x7f97cc1493f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.270+0000 7f97d2c00640 1 -- 172.21.15.28:0/2096680807 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc066c20 con 0x7f97cc0c7560 2024-09-13T16:46:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.270+0000 7f97c3400640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc0c7560 0x7f97cc1493f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.271+0000 7f97c3400640 1 -- 172.21.15.28:0/2096680807 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97cc067aa0 con 0x7f97cc0c7560 2024-09-13T16:46:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.271+0000 7f97c3400640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc0c7560 0x7f97cc1493f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f97cc1f0de0 tx=0x7f97b4001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.271+0000 7f97c8c00640 1 -- 172.21.15.28:0/2096680807 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f97bc039040 con 0x7f97cc06dcb0 2024-09-13T16:46:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.272+0000 7f97c8c00640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f97a806ac00 0x7f97a806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.272+0000 7f97c8c00640 1 -- 172.21.15.28:0/2096680807 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f97bc0a9cb0 con 0x7f97cc06dcb0 2024-09-13T16:46:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.272+0000 7f97d0c00640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f97a806ac00 0x7f97a806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.272+0000 7f97c8c00640 1 -- 172.21.15.28:0/2096680807 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b400fac0 con 0x7f97cc0c7560 2024-09-13T16:46:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.272+0000 7f97c8c00640 1 -- 172.21.15.28:0/2096680807 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f97b400fc20 con 0x7f97cc0c7560 2024-09-13T16:46:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.272+0000 7f97c8c00640 1 -- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc0c7560 msgr2=0x7f97cc1493f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.272+0000 7f97c8c00640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc0c7560 0x7f97cc1493f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f97cc1f0de0 tx=0x7f97b4001e70 comp rx=0 tx=0).stop 2024-09-13T16:46:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.272+0000 7f97c8c00640 1 -- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc0c7560 msgr2=0x7f97cc1493f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:46:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.272+0000 7f97c8c00640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc0c7560 0x7f97cc1493f0 secure :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f97cc1f0de0 tx=0x7f97b4001e70 comp rx=0 tx=0).stop 2024-09-13T16:46:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.273+0000 7f97d0c00640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f97a806ac00 0x7f97a806d0c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f97cc06e110 tx=0x7f97c400b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.279+0000 7f97d2c00640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc1521e0 0x7f97cc152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.279+0000 7f97d2c00640 1 -- 172.21.15.28:0/2096680807 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc0c2120 con 0x7f97cc1521e0 2024-09-13T16:46:11.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.279+0000 7f97c3400640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc1521e0 0x7f97cc152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.279+0000 7f97c3400640 1 -- 172.21.15.28:0/2096680807 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f97cc0c5220 con 0x7f97cc1521e0 2024-09-13T16:46:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.280+0000 7f97c3400640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc1521e0 0x7f97cc152d30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f97cc073f70 tx=0x7f97b4001080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.280+0000 7f97c8c00640 1 -- 172.21.15.28:0/2096680807 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b4023070 con 0x7f97cc1521e0 2024-09-13T16:46:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.280+0000 7f97c8c00640 1 -- 172.21.15.28:0/2096680807 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4457 (secure 0 0 0) 0x7f97b401c390 con 0x7f97cc1521e0 2024-09-13T16:46:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.280+0000 7f97c8c00640 1 -- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc1521e0 msgr2=0x7f97cc152d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.280+0000 7f97c8c00640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc1521e0 0x7f97cc152d30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f97cc073f70 tx=0x7f97b4001080 comp rx=0 tx=0).stop 2024-09-13T16:46:11.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.280+0000 7f97c8c00640 1 -- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc1521e0 msgr2=0x7f97cc152d30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:46:11.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.280+0000 7f97c8c00640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97cc1521e0 0x7f97cc152d30 secure :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f97cc073f70 tx=0x7f97b4001080 comp rx=0 tx=0).stop 2024-09-13T16:46:11.284 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:46:11.284 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:46:11.284 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:46:11.284 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:46:11.284 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 16, 2024-09-13T16:46:11.284 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:46:11.284 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:46:11.284 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:46:11.284 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:46:11.284 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:11.285 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 0, 2024-09-13T16:46:11.285 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:46:11.285 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:46:11.285 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:46:11.285 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:11.285 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:11.285 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:11.285 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:11.285 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:11.285 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:11.285 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:11.285 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:11.285 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:11.285 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:11.286 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:11.286 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:46:11.286 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:46:11.286 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:11.289 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:11.289 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:11.289 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:11.289 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:11.289 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:11.289 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:11.290 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:11.290 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:11.290 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:11.290 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.290 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:46:11.290 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:46:11.290 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:46:11.290 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:46:11.290 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:46:11.291 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:46:11.291 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:46:11.291 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:46:11.291 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:46:11.291 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:46:11.291 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:46:11.291 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:46:11.291 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:46:11.291 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:46:11.291 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:46:11.291 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:46:11.291 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:46:11.292 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:11.292 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:11.292 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:11.292 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:11.292 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:11.292 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:11.292 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:11.292 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:11.292 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:11.292 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:11.292 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:11.292 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:46:11.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.292 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:46:11.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.293 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:46:11.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:46:11.293 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:11.293 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:11.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.293 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:11.293 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:46:11.293 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.293 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:11.293 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:46:11.294 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.294 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.294 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.294 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:46:11.294 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:46:11.294 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:11.294 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:11.294 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:11.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.294 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.295 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:46:11.295 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:46:11.295 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:11.295 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:11.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.295 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:11.296 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:46:11.296 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.296 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.296 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:11.296 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:46:11.296 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.296 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.296 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.297 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:46:11.297 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:46:11.297 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:11.297 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:11.297 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:11.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.297 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:46:11.297 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:46:11.297 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:11.297 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:11.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:11.297 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:46:11.298 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.298 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:11.298 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:46:11.298 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.298 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.298 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.298 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:46:11.298 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:46:11.299 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:11.299 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:11.299 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:11.299 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.299 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.299 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:46:11.299 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:46:11.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.299 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:11.299 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:11.299 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:46:11.299 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.299 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:11.299 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:46:11.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.300 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:11.300 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:11.300 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:46:11.300 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.300 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.300 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:46:11.300 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:46:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.281+0000 7f97d2c00640 1 -- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f97a806ac00 msgr2=0x7f97a806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.281+0000 7f97d2c00640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f97a806ac00 0x7f97a806d0c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f97cc06e110 tx=0x7f97c400b040 comp rx=0 tx=0).stop 2024-09-13T16:46:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.281+0000 7f97d2c00640 1 -- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97cc06dcb0 msgr2=0x7f97cc1585e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.281+0000 7f97d2c00640 1 --2- 172.21.15.28:0/2096680807 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97cc06dcb0 0x7f97cc1585e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f97bc031000 tx=0x7f97bc0046c0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.281+0000 7f97d0c00640 1 -- 172.21.15.28:0/2096680807 reap_dead start 2024-09-13T16:46:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.281+0000 7f97d0c00640 1 -- 172.21.15.28:0/2096680807 reap_dead start 2024-09-13T16:46:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.281+0000 7f97d2c00640 1 -- 172.21.15.28:0/2096680807 shutdown_connections 2024-09-13T16:46:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.281+0000 7f97d2c00640 1 -- 172.21.15.28:0/2096680807 >> 172.21.15.28:0/2096680807 conn(0x7f97cc0ca6b0 msgr2=0x7f97cc14cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.281+0000 7f97d2c00640 1 -- 172.21.15.28:0/2096680807 shutdown_connections 2024-09-13T16:46:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.281+0000 7f97d2c00640 1 -- 172.21.15.28:0/2096680807 wait complete. 2024-09-13T16:46:11.302 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:46:11.303 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-13T16:46:11.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T16:46:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.457+0000 7ff29f000640 1 Processor -- start 2024-09-13T16:46:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.457+0000 7ff29f000640 1 -- start start 2024-09-13T16:46:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.459+0000 7ff29f000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff298152d50 0x7ff2980c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.459+0000 7ff29f000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff2980c5740 0x7ff29806fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.459+0000 7ff29f000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2980c6cf0 con 0x7ff2980c5340 2024-09-13T16:46:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.459+0000 7ff29f000640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff29815c080 con 0x7ff2980c5740 2024-09-13T16:46:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.459+0000 7ff29f000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29815cbc0 con 0x7ff298152d50 2024-09-13T16:46:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.459+0000 7ff28fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff2980c5340 0x7ff29806d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57086/0 (socket says 172.21.15.28:57086) 2024-09-13T16:46:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.459+0000 7ff28fe00640 1 -- 172.21.15.28:0/1004354506 learned_addr learned my addr 172.21.15.28:0/1004354506 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:46:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.459+0000 7ff297e00640 1 --2- 172.21.15.28:0/1004354506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff298152d50 0x7ff2980c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.459+0000 7ff29d000640 1 --2- 172.21.15.28:0/1004354506 >> v2:172.21.15.191:3300/0 conn(0x7ff2980c5740 0x7ff29806fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.459+0000 7ff297400640 1 -- 172.21.15.28:0/1004354506 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585860484 0 0) 0x7ff280001110 con 0x7ff2980c5340 2024-09-13T16:46:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.459+0000 7ff297400640 1 -- 172.21.15.28:0/1004354506 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff27c003520 con 0x7ff2980c5340 2024-09-13T16:46:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.460+0000 7ff297e00640 1 -- 172.21.15.28:0/1004354506 >> v2:172.21.15.191:3300/0 conn(0x7ff2980c5740 msgr2=0x7ff29806fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:46:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.460+0000 7ff297e00640 1 --2- 172.21.15.28:0/1004354506 >> v2:172.21.15.191:3300/0 conn(0x7ff2980c5740 0x7ff29806fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.460+0000 7ff297e00640 1 -- 172.21.15.28:0/1004354506 >> v1:172.21.15.28:6789/0 conn(0x7ff2980c5340 legacy=0x7ff29806d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.460+0000 7ff297e00640 1 -- 172.21.15.28:0/1004354506 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2980c5d10 con 0x7ff298152d50 2024-09-13T16:46:11.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.460+0000 7ff29d000640 1 --2- 172.21.15.28:0/1004354506 >> v2:172.21.15.191:3300/0 conn(0x7ff2980c5740 0x7ff29806fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:46:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.461+0000 7ff297e00640 1 --2- 172.21.15.28:0/1004354506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff298152d50 0x7ff2980c67b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff2880097a0 tx=0x7ff28802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c13a53cab3c5916 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.461+0000 7ff297400640 1 -- 172.21.15.28:0/1004354506 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288032440 con 0x7ff298152d50 2024-09-13T16:46:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.461+0000 7ff297400640 1 -- 172.21.15.28:0/1004354506 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2880325a0 con 0x7ff298152d50 2024-09-13T16:46:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.461+0000 7ff297400640 1 -- 172.21.15.28:0/1004354506 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288032930 con 0x7ff298152d50 2024-09-13T16:46:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.461+0000 7ff29f000640 1 -- 172.21.15.28:0/1004354506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff298152d50 msgr2=0x7ff2980c67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.461+0000 7ff29f000640 1 --2- 172.21.15.28:0/1004354506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff298152d50 0x7ff2980c67b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff2880097a0 tx=0x7ff28802a210 comp rx=0 tx=0).stop 2024-09-13T16:46:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.461+0000 7ff29f000640 1 -- 172.21.15.28:0/1004354506 shutdown_connections 2024-09-13T16:46:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.461+0000 7ff29f000640 1 --2- 172.21.15.28:0/1004354506 >> v2:172.21.15.191:3300/0 conn(0x7ff2980c5740 0x7ff29806fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.461+0000 7ff29f000640 1 --2- 172.21.15.28:0/1004354506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff298152d50 0x7ff2980c67b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.461+0000 7ff29f000640 1 -- 172.21.15.28:0/1004354506 >> 172.21.15.28:0/1004354506 conn(0x7ff2980ca6b0 msgr2=0x7ff2980caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.461+0000 7ff29f000640 1 -- 172.21.15.28:0/1004354506 shutdown_connections 2024-09-13T16:46:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.461+0000 7ff29f000640 1 -- 172.21.15.28:0/1004354506 wait complete. 2024-09-13T16:46:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.463+0000 7ff29f000640 1 Processor -- start 2024-09-13T16:46:11.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.463+0000 7ff29f000640 1 -- start start 2024-09-13T16:46:11.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.464+0000 7ff29f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2980c5340 0x7ff298148530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.464+0000 7ff29f000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2980c5740 0x7ff298146890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.464+0000 7ff29f000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff298152d50 0x7ff29815ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.464+0000 7ff29f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29815a2d0 con 0x7ff2980c5340 2024-09-13T16:46:11.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.464+0000 7ff29f000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff298145280 con 0x7ff298152d50 2024-09-13T16:46:11.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.464+0000 7ff29f000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff298159d10 con 0x7ff2980c5740 2024-09-13T16:46:11.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.464+0000 7ff297e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2980c5340 0x7ff298148530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.464+0000 7ff297e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2980c5340 0x7ff298148530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51618/0 (socket says 172.21.15.28:51618) 2024-09-13T16:46:11.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.464+0000 7ff297e00640 1 -- 172.21.15.28:0/2847857152 learned_addr learned my addr 172.21.15.28:0/2847857152 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:11.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.464+0000 7ff29d000640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff298152d50 0x7ff29815ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.464+0000 7ff28fe00640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2980c5740 0x7ff298146890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.465+0000 7ff297e00640 1 -- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2980c5740 msgr2=0x7ff298146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.465+0000 7ff297e00640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2980c5740 0x7ff298146890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.465+0000 7ff297e00640 1 -- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff298152d50 msgr2=0x7ff29815ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.465+0000 7ff297e00640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff298152d50 0x7ff29815ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.465+0000 7ff297e00640 1 -- 172.21.15.28:0/2847857152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff290009a70 con 0x7ff2980c5340 2024-09-13T16:46:11.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.465+0000 7ff28fe00640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff2980c5740 0x7ff298146890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:46:11.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.465+0000 7ff297e00640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2980c5340 0x7ff298148530 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff288000c00 tx=0x7ff2880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.465+0000 7ff294c00640 1 -- 172.21.15.28:0/2847857152 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28803d830 con 0x7ff2980c5340 2024-09-13T16:46:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.465+0000 7ff294c00640 1 -- 172.21.15.28:0/2847857152 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28803d990 con 0x7ff2980c5340 2024-09-13T16:46:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.465+0000 7ff29f000640 1 -- 172.21.15.28:0/2847857152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff288009400 con 0x7ff2980c5340 2024-09-13T16:46:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.465+0000 7ff294c00640 1 -- 172.21.15.28:0/2847857152 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28803dd40 con 0x7ff2980c5340 2024-09-13T16:46:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.465+0000 7ff29f000640 1 -- 172.21.15.28:0/2847857152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff298145770 con 0x7ff2980c5340 2024-09-13T16:46:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.466+0000 7ff29f000640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff298145c90 0x7ff29815b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.466+0000 7ff29f000640 1 -- 172.21.15.28:0/2847857152 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29815a190 con 0x7ff298145c90 2024-09-13T16:46:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.466+0000 7ff28fe00640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff298145c90 0x7ff29815b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.467+0000 7ff28fe00640 1 -- 172.21.15.28:0/2847857152 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2980c50b0 con 0x7ff298145c90 2024-09-13T16:46:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.467+0000 7ff28fe00640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff298145c90 0x7ff29815b0c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff280001270 tx=0x7ff28000eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.467+0000 7ff294c00640 1 -- 172.21.15.28:0/2847857152 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28000ee40 con 0x7ff298145c90 2024-09-13T16:46:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.468+0000 7ff294c00640 1 -- 172.21.15.28:0/2847857152 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff288039040 con 0x7ff2980c5340 2024-09-13T16:46:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.468+0000 7ff294c00640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff27406aca0 0x7ff27406d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.468+0000 7ff294c00640 1 -- 172.21.15.28:0/2847857152 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff2880a9eb0 con 0x7ff2980c5340 2024-09-13T16:46:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.469+0000 7ff294c00640 1 -- 172.21.15.28:0/2847857152 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7ff280005aa0 con 0x7ff298145c90 2024-09-13T16:46:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.469+0000 7ff29d000640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff27406aca0 0x7ff27406d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.469+0000 7ff294c00640 1 -- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff298145c90 msgr2=0x7ff29815b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.469+0000 7ff294c00640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff298145c90 0x7ff29815b0c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff280001270 tx=0x7ff28000eb90 comp rx=0 tx=0).stop 2024-09-13T16:46:11.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.469+0000 7ff294c00640 1 -- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff298145c90 msgr2=0x7ff29815b0c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:46:11.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.469+0000 7ff294c00640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff298145c90 0x7ff29815b0c0 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff280001270 tx=0x7ff28000eb90 comp rx=0 tx=0).stop 2024-09-13T16:46:11.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.469+0000 7ff29d000640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff27406aca0 0x7ff27406d160 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff290011580 tx=0x7ff29000b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.475+0000 7ff29f000640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff29815b840 0x7ff2980c2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.475+0000 7ff29f000640 1 -- 172.21.15.28:0/2847857152 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff298066c20 con 0x7ff29815b840 2024-09-13T16:46:11.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.476+0000 7ff28fe00640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff29815b840 0x7ff2980c2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.476+0000 7ff28fe00640 1 -- 172.21.15.28:0/2847857152 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff29815c850 con 0x7ff29815b840 2024-09-13T16:46:11.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.476+0000 7ff28fe00640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff29815b840 0x7ff2980c2780 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff2980c60f0 tx=0x7ff2800012e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.477+0000 7ff294c00640 1 -- 172.21.15.28:0/2847857152 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28001d9d0 con 0x7ff29815b840 2024-09-13T16:46:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.477+0000 7ff294c00640 1 -- 172.21.15.28:0/2847857152 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4789 (secure 0 0 0) 0x7ff28001db30 con 0x7ff29815b840 2024-09-13T16:46:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.477+0000 7ff294c00640 1 -- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff29815b840 msgr2=0x7ff2980c2780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.477+0000 7ff294c00640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff29815b840 0x7ff2980c2780 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff2980c60f0 tx=0x7ff2800012e0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.477+0000 7ff294c00640 1 -- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff29815b840 msgr2=0x7ff2980c2780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:46:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.477+0000 7ff294c00640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff29815b840 0x7ff2980c2780 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff2980c60f0 tx=0x7ff2800012e0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.479 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:46:11.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:46:11.480 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:46:11.480 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:46:11.480 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 16, 2024-09-13T16:46:11.480 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:46:11.480 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:46:11.480 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:46:11.480 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:46:11.480 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:11.480 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 0, 2024-09-13T16:46:11.480 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:46:11.480 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:46:11.481 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:46:11.481 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:11.481 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:11.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:11.481 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:11.481 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:11.481 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:11.481 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:11.481 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:11.482 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:11.482 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:11.482 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:11.482 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:46:11.482 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:46:11.482 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:11.482 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:11.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:11.483 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:11.483 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:11.483 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:11.483 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:11.483 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:11.483 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:11.484 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:11.484 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.484 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:46:11.484 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:46:11.484 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:46:11.484 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:46:11.484 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:46:11.484 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:46:11.484 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:46:11.485 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:46:11.485 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:46:11.485 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:46:11.485 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:46:11.485 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:46:11.485 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:46:11.485 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:46:11.485 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:46:11.485 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:46:11.485 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:46:11.485 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:11.486 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:11.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:11.486 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:11.486 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:11.486 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:11.486 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:11.486 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:11.486 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:11.486 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:11.486 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:11.486 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:46:11.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:46:11.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.487 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:46:11.487 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:46:11.487 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:11.487 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:11.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.487 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:11.487 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:46:11.487 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.487 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:11.488 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:46:11.488 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.488 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.488 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:46:11.488 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:46:11.488 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:11.488 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:11.488 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:11.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.489 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:46:11.489 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:46:11.489 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:11.489 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:11.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.493 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:11.493 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:46:11.493 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.494 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:11.494 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:46:11.494 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.494 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.494 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.494 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:46:11.494 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:46:11.494 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:11.494 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:11.494 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:11.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.495 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:46:11.495 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:46:11.495 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:11.495 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:11.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.495 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:11.495 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:46:11.495 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.495 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:11.495 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:46:11.496 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:11.496 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.496 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.496 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:46:11.496 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:46:11.496 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:11.496 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:11.496 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:11.496 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.496 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.496 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:46:11.497 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:46:11.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.497 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:11.497 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:11.497 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:46:11.497 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.497 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:11.497 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:46:11.497 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.497 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:11.497 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:11.498 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:46:11.498 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.498 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:11.498 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:46:11.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.498 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:11.498 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:11.498 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:46:11.498 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.498 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:11.498 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:46:11.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:11.499 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:11.499 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:11.499 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:46:11.499 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:11.499 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:11.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:11.499 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:46:11.499 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:46:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.477+0000 7ff29f000640 1 -- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff27406aca0 msgr2=0x7ff27406d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.477+0000 7ff29f000640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff27406aca0 0x7ff27406d160 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff290011580 tx=0x7ff29000b040 comp rx=0 tx=0).stop 2024-09-13T16:46:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.477+0000 7ff29f000640 1 -- 172.21.15.28:0/2847857152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2980c5340 msgr2=0x7ff298148530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.477+0000 7ff29f000640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2980c5340 0x7ff298148530 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff288000c00 tx=0x7ff2880046c0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.477+0000 7ff29d000640 1 -- 172.21.15.28:0/2847857152 reap_dead start 2024-09-13T16:46:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.477+0000 7ff29f000640 1 -- 172.21.15.28:0/2847857152 shutdown_connections 2024-09-13T16:46:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.477+0000 7ff29f000640 1 --2- 172.21.15.28:0/2847857152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2980c5340 0x7ff298148530 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.477+0000 7ff29f000640 1 -- 172.21.15.28:0/2847857152 >> 172.21.15.28:0/2847857152 conn(0x7ff2980ca6b0 msgr2=0x7ff298152060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.477+0000 7ff29f000640 1 -- 172.21.15.28:0/2847857152 shutdown_connections 2024-09-13T16:46:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.478+0000 7ff29f000640 1 -- 172.21.15.28:0/2847857152 wait complete. 2024-09-13T16:46:11.502 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:46:11.502 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-13T16:46:11.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-13T16:46:11.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.657+0000 7f7f05800640 1 Processor -- start 2024-09-13T16:46:11.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.657+0000 7f7f05800640 1 -- start start 2024-09-13T16:46:11.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.658+0000 7f7f05800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7f00150ca0 0x7f7f001568e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.658+0000 7f7f05800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f000c4d60 0x7f7f000c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.658+0000 7f7f05800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f00151080 con 0x7f7f000c2250 2024-09-13T16:46:11.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.658+0000 7f7f05800640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f000c2670 con 0x7f7f00150ca0 2024-09-13T16:46:11.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.658+0000 7f7f05800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0015ae40 con 0x7f7f000c4d60 2024-09-13T16:46:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.659+0000 7f7efe000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7f000c2250 0x7f7f0015a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57090/0 (socket says 172.21.15.28:57090) 2024-09-13T16:46:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.659+0000 7f7efe000640 1 -- 172.21.15.28:0/3297813039 learned_addr learned my addr 172.21.15.28:0/3297813039 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:46:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.659+0000 7f7efea00640 1 --2- 172.21.15.28:0/3297813039 >> v2:172.21.15.191:3300/0 conn(0x7f7f00150ca0 0x7f7f001568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.659+0000 7f7eff400640 1 --2- 172.21.15.28:0/3297813039 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f000c4d60 0x7f7f000c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.659+0000 7f7efd600640 1 -- 172.21.15.28:0/3297813039 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270145787 0 0) 0x7f7f00151080 con 0x7f7f000c2250 2024-09-13T16:46:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.659+0000 7f7efd600640 1 -- 172.21.15.28:0/3297813039 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ee4003520 con 0x7f7f000c2250 2024-09-13T16:46:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.659+0000 7f7eff400640 1 -- 172.21.15.28:0/3297813039 >> v2:172.21.15.191:3300/0 conn(0x7f7f00150ca0 msgr2=0x7f7f001568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:46:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.659+0000 7f7eff400640 1 --2- 172.21.15.28:0/3297813039 >> v2:172.21.15.191:3300/0 conn(0x7f7f00150ca0 0x7f7f001568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.659+0000 7f7eff400640 1 -- 172.21.15.28:0/3297813039 >> v1:172.21.15.28:6789/0 conn(0x7f7f000c2250 legacy=0x7f7f0015a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.659+0000 7f7eff400640 1 -- 172.21.15.28:0/3297813039 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0006a990 con 0x7f7f000c4d60 2024-09-13T16:46:11.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.659+0000 7f7efea00640 1 --2- 172.21.15.28:0/3297813039 >> v2:172.21.15.191:3300/0 conn(0x7f7f00150ca0 0x7f7f001568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:46:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.660+0000 7f7eff400640 1 --2- 172.21.15.28:0/3297813039 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f000c4d60 0x7f7f000c51d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8009d90 tx=0x7f7ef802a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b717c674177acb1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.660+0000 7f7efd600640 1 -- 172.21.15.28:0/3297813039 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8031600 con 0x7f7f000c4d60 2024-09-13T16:46:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.660+0000 7f7efd600640 1 -- 172.21.15.28:0/3297813039 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef802d050 con 0x7f7f000c4d60 2024-09-13T16:46:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.660+0000 7f7efd600640 1 -- 172.21.15.28:0/3297813039 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8031980 con 0x7f7f000c4d60 2024-09-13T16:46:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.661+0000 7f7f05800640 1 -- 172.21.15.28:0/3297813039 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f000c4d60 msgr2=0x7f7f000c51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.661+0000 7f7f05800640 1 --2- 172.21.15.28:0/3297813039 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f000c4d60 0x7f7f000c51d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8009d90 tx=0x7f7ef802a5d0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.661+0000 7f7f05800640 1 -- 172.21.15.28:0/3297813039 shutdown_connections 2024-09-13T16:46:11.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.661+0000 7f7f05800640 1 --2- 172.21.15.28:0/3297813039 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f000c4d60 0x7f7f000c51d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.661+0000 7f7f05800640 1 --2- 172.21.15.28:0/3297813039 >> v2:172.21.15.191:3300/0 conn(0x7f7f00150ca0 0x7f7f001568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.661+0000 7f7f05800640 1 -- 172.21.15.28:0/3297813039 >> 172.21.15.28:0/3297813039 conn(0x7f7f00066040 msgr2=0x7f7f00069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:11.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.661+0000 7f7f05800640 1 -- 172.21.15.28:0/3297813039 shutdown_connections 2024-09-13T16:46:11.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.661+0000 7f7f05800640 1 -- 172.21.15.28:0/3297813039 wait complete. 2024-09-13T16:46:11.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.663+0000 7f7f05800640 1 Processor -- start 2024-09-13T16:46:11.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.663+0000 7f7f05800640 1 -- start start 2024-09-13T16:46:11.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.664+0000 7f7f05800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f000c2250 0x7f7f001ec5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.664+0000 7f7f05800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f000c4d60 0x7f7f00158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.664+0000 7f7f05800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f00150ca0 0x7f7f001e0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.664+0000 7f7f05800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f001e1b90 con 0x7f7f00150ca0 2024-09-13T16:46:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.664+0000 7f7f05800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f001e1e50 con 0x7f7f000c2250 2024-09-13T16:46:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.664+0000 7f7f05800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f001e2130 con 0x7f7f000c4d60 2024-09-13T16:46:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.664+0000 7f7eff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f00150ca0 0x7f7f001e0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.664+0000 7f7eff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f00150ca0 0x7f7f001e0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51626/0 (socket says 172.21.15.28:51626) 2024-09-13T16:46:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.664+0000 7f7eff400640 1 -- 172.21.15.28:0/679728950 learned_addr learned my addr 172.21.15.28:0/679728950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.664+0000 7f7efea00640 1 --2- 172.21.15.28:0/679728950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f000c2250 0x7f7f001ec5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.664+0000 7f7efe000640 1 --2- 172.21.15.28:0/679728950 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f000c4d60 0x7f7f00158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.664+0000 7f7eff400640 1 -- 172.21.15.28:0/679728950 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f000c4d60 msgr2=0x7f7f00158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.664+0000 7f7eff400640 1 --2- 172.21.15.28:0/679728950 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f000c4d60 0x7f7f00158e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.664+0000 7f7eff400640 1 -- 172.21.15.28:0/679728950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f000c2250 msgr2=0x7f7f001ec5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.664+0000 7f7eff400640 1 --2- 172.21.15.28:0/679728950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f000c2250 0x7f7f001ec5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.664+0000 7f7eff400640 1 -- 172.21.15.28:0/679728950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef0009400 con 0x7f7f00150ca0 2024-09-13T16:46:11.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.665+0000 7f7eff400640 1 --2- 172.21.15.28:0/679728950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f00150ca0 0x7f7f001e0530 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7ef802abc0 tx=0x7f7ef8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.665+0000 7f7ef6a00640 1 -- 172.21.15.28:0/679728950 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef803c830 con 0x7f7f00150ca0 2024-09-13T16:46:11.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.665+0000 7f7ef6a00640 1 -- 172.21.15.28:0/679728950 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef803c990 con 0x7f7f00150ca0 2024-09-13T16:46:11.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.665+0000 7f7f05800640 1 -- 172.21.15.28:0/679728950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef8009a70 con 0x7f7f00150ca0 2024-09-13T16:46:11.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.665+0000 7f7ef6a00640 1 -- 172.21.15.28:0/679728950 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef803cd20 con 0x7f7f00150ca0 2024-09-13T16:46:11.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.665+0000 7f7f05800640 1 -- 172.21.15.28:0/679728950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f001e0010 con 0x7f7f00150ca0 2024-09-13T16:46:11.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.666+0000 7f7f05800640 1 -- 172.21.15.28:0/679728950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f001e0230 con 0x7f7f00150ca0 2024-09-13T16:46:11.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.667+0000 7f7ef6a00640 1 -- 172.21.15.28:0/679728950 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7ef803b3b0 con 0x7f7f00150ca0 2024-09-13T16:46:11.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.668+0000 7f7ef6a00640 1 --2- 172.21.15.28:0/679728950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ee006ac00 0x7f7ee006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:11.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.668+0000 7f7ef6a00640 1 -- 172.21.15.28:0/679728950 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7ef80aac90 con 0x7f7f00150ca0 2024-09-13T16:46:11.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.668+0000 7f7efea00640 1 --2- 172.21.15.28:0/679728950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ee006ac00 0x7f7ee006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:11.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.669+0000 7f7efea00640 1 --2- 172.21.15.28:0/679728950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ee006ac00 0x7f7ee006d0c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009cd0 tx=0x7f7ef0009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:11.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.672+0000 7f7ef6a00640 1 -- 172.21.15.28:0/679728950 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7ef8075e00 con 0x7f7f00150ca0 2024-09-13T16:46:11.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.918+0000 7f7f05800640 1 -- 172.21.15.28:0/679728950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f7f0015a210 con 0x7f7f00150ca0 2024-09-13T16:46:11.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.918+0000 7f7ef6a00640 1 -- 172.21.15.28:0/679728950 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f7ef8009450 con 0x7f7f00150ca0 2024-09-13T16:46:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.920+0000 7f7f05800640 1 -- 172.21.15.28:0/679728950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ee006ac00 msgr2=0x7f7ee006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.920+0000 7f7f05800640 1 --2- 172.21.15.28:0/679728950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ee006ac00 0x7f7ee006d0c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009cd0 tx=0x7f7ef0009190 comp rx=0 tx=0).stop 2024-09-13T16:46:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.920+0000 7f7f05800640 1 -- 172.21.15.28:0/679728950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f00150ca0 msgr2=0x7f7f001e0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.920+0000 7f7f05800640 1 --2- 172.21.15.28:0/679728950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f00150ca0 0x7f7f001e0530 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7ef802abc0 tx=0x7f7ef8004a40 comp rx=0 tx=0).stop 2024-09-13T16:46:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.920+0000 7f7f05800640 1 -- 172.21.15.28:0/679728950 shutdown_connections 2024-09-13T16:46:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.920+0000 7f7f05800640 1 --2- 172.21.15.28:0/679728950 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f7ee006ac00 0x7f7ee006d0c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.920+0000 7f7f05800640 1 --2- 172.21.15.28:0/679728950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f00150ca0 0x7f7f001e0530 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.920+0000 7f7f05800640 1 --2- 172.21.15.28:0/679728950 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7f000c4d60 0x7f7f00158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.920+0000 7f7f05800640 1 --2- 172.21.15.28:0/679728950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f000c2250 0x7f7f001ec5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.920+0000 7f7f05800640 1 -- 172.21.15.28:0/679728950 >> 172.21.15.28:0/679728950 conn(0x7f7f00066040 msgr2=0x7f7f0014ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.921+0000 7f7f05800640 1 -- 172.21.15.28:0/679728950 shutdown_connections 2024-09-13T16:46:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:11.921+0000 7f7f05800640 1 -- 172.21.15.28:0/679728950 wait complete. 2024-09-13T16:46:11.939 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-09-13T16:46:15.044 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:15.045 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:15.045 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:15.045 INFO:tasks.workunit.client.0.smithi028.stdout:iozone test complete. 2024-09-13T16:46:15.047 INFO:tasks.workunit.client.0.smithi028.stderr:+ iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-09-13T16:46:15.127 INFO:tasks.workunit.client.0.smithi028.stdout: Iozone: Performance Test of File I/O 2024-09-13T16:46:15.128 INFO:tasks.workunit.client.0.smithi028.stdout: Version $Revision: 3.494 $ 2024-09-13T16:46:15.128 INFO:tasks.workunit.client.0.smithi028.stdout: Compiled for 64 bit mode. 2024-09-13T16:46:15.128 INFO:tasks.workunit.client.0.smithi028.stdout: Build: linux-AMD64 2024-09-13T16:46:15.128 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:15.128 INFO:tasks.workunit.client.0.smithi028.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-09-13T16:46:15.128 INFO:tasks.workunit.client.0.smithi028.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-09-13T16:46:15.128 INFO:tasks.workunit.client.0.smithi028.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-09-13T16:46:15.128 INFO:tasks.workunit.client.0.smithi028.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-09-13T16:46:15.128 INFO:tasks.workunit.client.0.smithi028.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-09-13T16:46:15.128 INFO:tasks.workunit.client.0.smithi028.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-09-13T16:46:15.129 INFO:tasks.workunit.client.0.smithi028.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-09-13T16:46:15.129 INFO:tasks.workunit.client.0.smithi028.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-09-13T16:46:15.129 INFO:tasks.workunit.client.0.smithi028.stdout: Alexey Skidanov, Sudhir Kumar. 2024-09-13T16:46:15.129 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:15.129 INFO:tasks.workunit.client.0.smithi028.stdout: Run began: Fri Sep 13 16:46:15 2024 2024-09-13T16:46:15.129 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:15.129 INFO:tasks.workunit.client.0.smithi028.stdout: Include close in write timing 2024-09-13T16:46:15.129 INFO:tasks.workunit.client.0.smithi028.stdout: Include fsync in write timing 2024-09-13T16:46:15.129 INFO:tasks.workunit.client.0.smithi028.stdout: File size set to 1048576 kB 2024-09-13T16:46:15.129 INFO:tasks.workunit.client.0.smithi028.stdout: Record Size 1024 kB 2024-09-13T16:46:15.129 INFO:tasks.workunit.client.0.smithi028.stdout: Command line used: iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-09-13T16:46:15.129 INFO:tasks.workunit.client.0.smithi028.stdout: Output is in kBytes/sec 2024-09-13T16:46:15.129 INFO:tasks.workunit.client.0.smithi028.stdout: Time Resolution = 0.000001 seconds. 2024-09-13T16:46:15.130 INFO:tasks.workunit.client.0.smithi028.stdout: Processor cache size set to 1024 kBytes. 2024-09-13T16:46:15.130 INFO:tasks.workunit.client.0.smithi028.stdout: Processor cache line size set to 32 bytes. 2024-09-13T16:46:15.130 INFO:tasks.workunit.client.0.smithi028.stdout: File stride size set to 17 * record size. 2024-09-13T16:46:15.130 INFO:tasks.workunit.client.0.smithi028.stdout: Throughput test with 1 process 2024-09-13T16:46:15.130 INFO:tasks.workunit.client.0.smithi028.stdout: Each process writes a 1048576 kByte file in 1024 kByte records 2024-09-13T16:46:18.514 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:18.514 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 initial writers = 313097.47 kB/sec 2024-09-13T16:46:18.514 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 initial writers = 313025.24 kB/sec 2024-09-13T16:46:18.514 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 313097.47 kB/sec 2024-09-13T16:46:18.514 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 313097.47 kB/sec 2024-09-13T16:46:18.515 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 313097.47 kB/sec 2024-09-13T16:46:18.515 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 1048576.00 kB 2024-09-13T16:46:21.940 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T16:46:21.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:46:22.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.185+0000 7f6e0e200640 1 Processor -- start 2024-09-13T16:46:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.185+0000 7f6e0e200640 1 -- start start 2024-09-13T16:46:22.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.188+0000 7f6e0e200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6e0814f6d0 0x7f6e080c6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:22.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.188+0000 7f6e0e200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6e0814fab0 0x7f6e080c18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:22.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.188+0000 7f6e0e200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e08155530 0x7f6e08155960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.188+0000 7f6e0e200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e080c2dc0 con 0x7f6e0814f6d0 2024-09-13T16:46:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.188+0000 7f6e0ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6e0814f6d0 0x7f6e080c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.188+0000 7f6e0e200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e080706b0 con 0x7f6e0814fab0 2024-09-13T16:46:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.188+0000 7f6e0e200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e08070f50 con 0x7f6e08155530 2024-09-13T16:46:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.188+0000 7f6e07e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6e0814fab0 0x7f6e080c18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.188+0000 7f6e07e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6e0814fab0 0x7f6e080c18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:47944/0 (socket says 172.21.15.28:47944) 2024-09-13T16:46:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.188+0000 7f6e07e00640 1 -- 172.21.15.28:0/2798622933 learned_addr learned my addr 172.21.15.28:0/2798622933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.188+0000 7f6e0d800640 1 --2- 172.21.15.28:0/2798622933 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e08155530 0x7f6e08155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.189+0000 7f6e07e00640 1 -- 172.21.15.28:0/2798622933 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e08155530 msgr2=0x7f6e08155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:46:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.189+0000 7f6e07e00640 1 --2- 172.21.15.28:0/2798622933 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e08155530 0x7f6e08155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.189+0000 7f6e07e00640 1 -- 172.21.15.28:0/2798622933 >> v2:172.21.15.28:3300/0 conn(0x7f6e0814f6d0 msgr2=0x7f6e080c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:46:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.189+0000 7f6e07e00640 1 --2- 172.21.15.28:0/2798622933 >> v2:172.21.15.28:3300/0 conn(0x7f6e0814f6d0 0x7f6e080c6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.189+0000 7f6e07e00640 1 -- 172.21.15.28:0/2798622933 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e080c41c0 con 0x7f6e0814fab0 2024-09-13T16:46:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.189+0000 7f6e0d800640 1 --2- 172.21.15.28:0/2798622933 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e08155530 0x7f6e08155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:46:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.190+0000 7f6e07e00640 1 --2- 172.21.15.28:0/2798622933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e0814fab0 0x7f6e080c18f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6e080c6000 tx=0x7f6df402a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66b91d586b8876b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.190+0000 7f6e07400640 1 -- 172.21.15.28:0/2798622933 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df4032440 con 0x7f6e0814fab0 2024-09-13T16:46:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.190+0000 7f6e07400640 1 -- 172.21.15.28:0/2798622933 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df40325a0 con 0x7f6e0814fab0 2024-09-13T16:46:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.190+0000 7f6e07400640 1 -- 172.21.15.28:0/2798622933 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df4032930 con 0x7f6e0814fab0 2024-09-13T16:46:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.191+0000 7f6e0e200640 1 -- 172.21.15.28:0/2798622933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e0814fab0 msgr2=0x7f6e080c18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.191+0000 7f6e0e200640 1 --2- 172.21.15.28:0/2798622933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e0814fab0 0x7f6e080c18f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6e080c6000 tx=0x7f6df402a270 comp rx=0 tx=0).stop 2024-09-13T16:46:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.191+0000 7f6e0e200640 1 -- 172.21.15.28:0/2798622933 shutdown_connections 2024-09-13T16:46:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.191+0000 7f6e0e200640 1 --2- 172.21.15.28:0/2798622933 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e08155530 0x7f6e08155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.191+0000 7f6e0e200640 1 --2- 172.21.15.28:0/2798622933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e0814fab0 0x7f6e080c18f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.191+0000 7f6e0e200640 1 --2- 172.21.15.28:0/2798622933 >> v2:172.21.15.28:3300/0 conn(0x7f6e0814f6d0 0x7f6e080c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.191+0000 7f6e0e200640 1 -- 172.21.15.28:0/2798622933 >> 172.21.15.28:0/2798622933 conn(0x7f6e08069e80 msgr2=0x7f6e0806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.191+0000 7f6e0e200640 1 -- 172.21.15.28:0/2798622933 shutdown_connections 2024-09-13T16:46:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.191+0000 7f6e0e200640 1 -- 172.21.15.28:0/2798622933 wait complete. 2024-09-13T16:46:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.193+0000 7f6e0e200640 1 Processor -- start 2024-09-13T16:46:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.193+0000 7f6e0e200640 1 -- start start 2024-09-13T16:46:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.194+0000 7f6e0e200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e0814f6d0 0x7f6e080c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.194+0000 7f6e0e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e08155530 0x7f6e081e12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.194+0000 7f6e0e200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e080c9560 0x7f6e081f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.194+0000 7f6e0e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e080c8830 con 0x7f6e08155530 2024-09-13T16:46:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.194+0000 7f6e0e200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e080c90d0 con 0x7f6e0814f6d0 2024-09-13T16:46:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.194+0000 7f6e0e200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e081e2400 con 0x7f6e080c9560 2024-09-13T16:46:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.194+0000 7f6e0ce00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e0814f6d0 0x7f6e080c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.194+0000 7f6e0ce00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e0814f6d0 0x7f6e080c9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:47952/0 (socket says 172.21.15.28:47952) 2024-09-13T16:46:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.194+0000 7f6e0ce00640 1 -- 172.21.15.28:0/526305250 learned_addr learned my addr 172.21.15.28:0/526305250 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.194+0000 7f6e0d800640 1 --2- 172.21.15.28:0/526305250 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e080c9560 0x7f6e081f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.194+0000 7f6e07e00640 1 --2- 172.21.15.28:0/526305250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e08155530 0x7f6e081e12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.194+0000 7f6e0ce00640 1 -- 172.21.15.28:0/526305250 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e080c9560 msgr2=0x7f6e081f1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.194+0000 7f6e0ce00640 1 --2- 172.21.15.28:0/526305250 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e080c9560 0x7f6e081f1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.194+0000 7f6e0ce00640 1 -- 172.21.15.28:0/526305250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e08155530 msgr2=0x7f6e081e12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.194+0000 7f6e0ce00640 1 --2- 172.21.15.28:0/526305250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e08155530 0x7f6e081e12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.194+0000 7f6e0ce00640 1 -- 172.21.15.28:0/526305250 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0000ba70 con 0x7f6e0814f6d0 2024-09-13T16:46:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.194+0000 7f6e0d800640 1 --2- 172.21.15.28:0/526305250 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e080c9560 0x7f6e081f1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:46:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.195+0000 7f6e0ce00640 1 --2- 172.21.15.28:0/526305250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e0814f6d0 0x7f6e080c9b00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00e8a0 tx=0x7f6dfc00ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.195+0000 7f6e04c00640 1 -- 172.21.15.28:0/526305250 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc00ee90 con 0x7f6e0814f6d0 2024-09-13T16:46:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.195+0000 7f6e04c00640 1 -- 172.21.15.28:0/526305250 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dfc00c940 con 0x7f6e0814f6d0 2024-09-13T16:46:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.195+0000 7f6e04c00640 1 -- 172.21.15.28:0/526305250 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc00ccf0 con 0x7f6e0814f6d0 2024-09-13T16:46:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.195+0000 7f6e0e200640 1 -- 172.21.15.28:0/526305250 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df40094d0 con 0x7f6e0814f6d0 2024-09-13T16:46:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.195+0000 7f6e0e200640 1 -- 172.21.15.28:0/526305250 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e081f2840 con 0x7f6e0814f6d0 2024-09-13T16:46:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.196+0000 7f6e0e200640 1 -- 172.21.15.28:0/526305250 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0814b260 con 0x7f6e0814f6d0 2024-09-13T16:46:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.201+0000 7f6e04c00640 1 -- 172.21.15.28:0/526305250 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6dfc013470 con 0x7f6e0814f6d0 2024-09-13T16:46:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.201+0000 7f6e04c00640 1 --2- 172.21.15.28:0/526305250 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6de806ac00 0x7f6de806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.202+0000 7f6e04c00640 1 -- 172.21.15.28:0/526305250 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6dfc089740 con 0x7f6e0814f6d0 2024-09-13T16:46:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.202+0000 7f6e07e00640 1 --2- 172.21.15.28:0/526305250 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6de806ac00 0x7f6de806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.203+0000 7f6e07e00640 1 --2- 172.21.15.28:0/526305250 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6de806ac00 0x7f6de806d0c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6df4034000 tx=0x7f6df402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.208+0000 7f6e04c00640 1 -- 172.21.15.28:0/526305250 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6dfc054950 con 0x7f6e0814f6d0 2024-09-13T16:46:22.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.687+0000 7f6e0e200640 1 -- 172.21.15.28:0/526305250 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6e08071e20 con 0x7f6e0814f6d0 2024-09-13T16:46:22.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.687+0000 7f6e04c00640 1 -- 172.21.15.28:0/526305250 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f6dfc017620 con 0x7f6e0814f6d0 2024-09-13T16:46:22.691 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:46:22.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.705+0000 7f6e0e200640 1 -- 172.21.15.28:0/526305250 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6de806ac00 msgr2=0x7f6de806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:22.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.705+0000 7f6e0e200640 1 --2- 172.21.15.28:0/526305250 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6de806ac00 0x7f6de806d0c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6df4034000 tx=0x7f6df402e000 comp rx=0 tx=0).stop 2024-09-13T16:46:22.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.705+0000 7f6e0e200640 1 -- 172.21.15.28:0/526305250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e0814f6d0 msgr2=0x7f6e080c9b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:22.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.705+0000 7f6e0e200640 1 --2- 172.21.15.28:0/526305250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e0814f6d0 0x7f6e080c9b00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00e8a0 tx=0x7f6dfc00ed60 comp rx=0 tx=0).stop 2024-09-13T16:46:22.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.705+0000 7f6e0e200640 1 -- 172.21.15.28:0/526305250 shutdown_connections 2024-09-13T16:46:22.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.705+0000 7f6e0e200640 1 --2- 172.21.15.28:0/526305250 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f6de806ac00 0x7f6de806d0c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.705+0000 7f6e0e200640 1 --2- 172.21.15.28:0/526305250 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e080c9560 0x7f6e081f1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.705+0000 7f6e0e200640 1 --2- 172.21.15.28:0/526305250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e08155530 0x7f6e081e12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.705+0000 7f6e0e200640 1 --2- 172.21.15.28:0/526305250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e0814f6d0 0x7f6e080c9b00 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00e8a0 tx=0x7f6dfc00ed60 comp rx=0 tx=0).stop 2024-09-13T16:46:22.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.705+0000 7f6e0e200640 1 -- 172.21.15.28:0/526305250 >> 172.21.15.28:0/526305250 conn(0x7f6e08069e80 msgr2=0x7f6e080c12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:22.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.705+0000 7f6e0e200640 1 -- 172.21.15.28:0/526305250 shutdown_connections 2024-09-13T16:46:22.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.705+0000 7f6e0e200640 1 -- 172.21.15.28:0/526305250 wait complete. 2024-09-13T16:46:22.731 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T16:46:22.731 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-13T16:46:22.731 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-13T16:46:22.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T16:46:22.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.964+0000 7fd50a000640 1 Processor -- start 2024-09-13T16:46:22.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.965+0000 7fd50a000640 1 -- start start 2024-09-13T16:46:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.966+0000 7fd50a000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd5040ce5e0 0x7fd5041595a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.966+0000 7fd50a000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5040c94d0 0x7fd5040c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.966+0000 7fd50a000640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd504159f90 con 0x7fd5040ce9c0 2024-09-13T16:46:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.966+0000 7fd50a000640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd504159ae0 con 0x7fd5040ce5e0 2024-09-13T16:46:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.966+0000 7fd50a000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5040c6900 con 0x7fd5040c94d0 2024-09-13T16:46:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.967+0000 7fd508c00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd5040ce5e0 0x7fd5041595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.967+0000 7fd503e00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fd5040ce9c0 0x7fd50415b590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:45510/0 (socket says 172.21.15.28:45510) 2024-09-13T16:46:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.967+0000 7fd508c00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd5040ce5e0 0x7fd5041595a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:47954/0 (socket says 172.21.15.28:47954) 2024-09-13T16:46:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.968+0000 7fd503e00640 1 -- 172.21.15.28:0/2215545162 learned_addr learned my addr 172.21.15.28:0/2215545162 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:46:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.968+0000 7fd509600640 1 --2- 172.21.15.28:0/2215545162 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5040c94d0 0x7fd5040c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.968+0000 7fd508c00640 1 -- 172.21.15.28:0/2215545162 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5040c94d0 msgr2=0x7fd5040c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:46:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.968+0000 7fd508c00640 1 --2- 172.21.15.28:0/2215545162 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5040c94d0 0x7fd5040c6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.968+0000 7fd508c00640 1 -- 172.21.15.28:0/2215545162 >> v1:172.21.15.191:6789/0 conn(0x7fd5040ce9c0 legacy=0x7fd50415b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.968+0000 7fd508c00640 1 -- 172.21.15.28:0/2215545162 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5040c5800 con 0x7fd5040ce5e0 2024-09-13T16:46:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.968+0000 7fd509600640 1 --2- 172.21.15.28:0/2215545162 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5040c94d0 0x7fd5040c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:46:22.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.970+0000 7fd508c00640 1 --2- 172.21.15.28:0/2215545162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5040ce5e0 0x7fd5041595a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd4f80097a0 tx=0x7fd4f802a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f517f2b11a2087b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:22.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.971+0000 7fd503400640 1 -- 172.21.15.28:0/2215545162 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8032440 con 0x7fd5040ce5e0 2024-09-13T16:46:22.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.971+0000 7fd503400640 1 -- 172.21.15.28:0/2215545162 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f80325a0 con 0x7fd5040ce5e0 2024-09-13T16:46:22.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.971+0000 7fd503400640 1 -- 172.21.15.28:0/2215545162 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8032930 con 0x7fd5040ce5e0 2024-09-13T16:46:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.971+0000 7fd50a000640 1 -- 172.21.15.28:0/2215545162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5040ce5e0 msgr2=0x7fd5041595a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.971+0000 7fd50a000640 1 --2- 172.21.15.28:0/2215545162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5040ce5e0 0x7fd5041595a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd4f80097a0 tx=0x7fd4f802a210 comp rx=0 tx=0).stop 2024-09-13T16:46:22.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.972+0000 7fd50a000640 1 -- 172.21.15.28:0/2215545162 shutdown_connections 2024-09-13T16:46:22.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.972+0000 7fd50a000640 1 --2- 172.21.15.28:0/2215545162 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5040c94d0 0x7fd5040c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.972+0000 7fd50a000640 1 --2- 172.21.15.28:0/2215545162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5040ce5e0 0x7fd5041595a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.972+0000 7fd50a000640 1 -- 172.21.15.28:0/2215545162 >> 172.21.15.28:0/2215545162 conn(0x7fd50406c880 msgr2=0x7fd5040715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:22.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.972+0000 7fd50a000640 1 -- 172.21.15.28:0/2215545162 shutdown_connections 2024-09-13T16:46:22.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.972+0000 7fd50a000640 1 -- 172.21.15.28:0/2215545162 wait complete. 2024-09-13T16:46:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.974+0000 7fd50a000640 1 Processor -- start 2024-09-13T16:46:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.974+0000 7fd50a000640 1 -- start start 2024-09-13T16:46:22.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.975+0000 7fd50a000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5040c94d0 0x7fd5040c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:22.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.975+0000 7fd50a000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5040ce5e0 0x7fd5041ee340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.975+0000 7fd50a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5040ce9c0 0x7fd50414d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.975+0000 7fd50a000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5040c1670 con 0x7fd5040ce9c0 2024-09-13T16:46:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.975+0000 7fd50a000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50414e910 con 0x7fd5040c94d0 2024-09-13T16:46:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.975+0000 7fd50a000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5041f18b0 con 0x7fd5040ce5e0 2024-09-13T16:46:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.975+0000 7fd503e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5040ce5e0 0x7fd5041ee340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.975+0000 7fd503e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5040ce5e0 0x7fd5041ee340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:39706/0 (socket says 172.21.15.28:39706) 2024-09-13T16:46:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.975+0000 7fd503e00640 1 -- 172.21.15.28:0/3085988034 learned_addr learned my addr 172.21.15.28:0/3085988034 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.976+0000 7fd508c00640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5040c94d0 0x7fd5040c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.976+0000 7fd509600640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5040ce9c0 0x7fd50414d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.976+0000 7fd503e00640 1 -- 172.21.15.28:0/3085988034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5040c94d0 msgr2=0x7fd5040c36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.976+0000 7fd503e00640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5040c94d0 0x7fd5040c36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.976+0000 7fd503e00640 1 -- 172.21.15.28:0/3085988034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5040ce9c0 msgr2=0x7fd50414d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.976+0000 7fd503e00640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5040ce9c0 0x7fd50414d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.976+0000 7fd503e00640 1 -- 172.21.15.28:0/3085988034 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f8009400 con 0x7fd5040ce5e0 2024-09-13T16:46:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.976+0000 7fd508c00640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5040c94d0 0x7fd5040c36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:46:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.977+0000 7fd503e00640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5040ce5e0 0x7fd5041ee340 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000c480 tx=0x7fd4f000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.977+0000 7fd500c00640 1 -- 172.21.15.28:0/3085988034 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0005ce0 con 0x7fd5040ce5e0 2024-09-13T16:46:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.977+0000 7fd500c00640 1 -- 172.21.15.28:0/3085988034 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f0005e40 con 0x7fd5040ce5e0 2024-09-13T16:46:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.977+0000 7fd500c00640 1 -- 172.21.15.28:0/3085988034 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f00072f0 con 0x7fd5040ce5e0 2024-09-13T16:46:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.977+0000 7fd50a000640 1 -- 172.21.15.28:0/3085988034 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5041f2110 con 0x7fd5040ce5e0 2024-09-13T16:46:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.977+0000 7fd50a000640 1 -- 172.21.15.28:0/3085988034 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5041f1ac0 con 0x7fd5040ce5e0 2024-09-13T16:46:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.978+0000 7fd50a000640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504155640 0x7fd50414df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.978+0000 7fd50a000640 1 -- 172.21.15.28:0/3085988034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd504072160 con 0x7fd504155640 2024-09-13T16:46:22.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.979+0000 7fd508c00640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504155640 0x7fd50414df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:22.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.979+0000 7fd500c00640 1 -- 172.21.15.28:0/3085988034 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd4f0007510 con 0x7fd5040ce5e0 2024-09-13T16:46:22.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.980+0000 7fd500c00640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd4e406ac00 0x7fd4e406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:22.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.980+0000 7fd500c00640 1 -- 172.21.15.28:0/3085988034 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd4f00888e0 con 0x7fd5040ce5e0 2024-09-13T16:46:22.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.980+0000 7fd509600640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd4e406ac00 0x7fd4e406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:22.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.981+0000 7fd508c00640 1 -- 172.21.15.28:0/3085988034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5040c5cf0 con 0x7fd504155640 2024-09-13T16:46:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.981+0000 7fd509600640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd4e406ac00 0x7fd4e406d0c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd5040c4070 tx=0x7fd4fc013a20 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.981+0000 7fd508c00640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504155640 0x7fd50414df00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd5040c4fe0 tx=0x7fd4f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.981+0000 7fd500c00640 1 -- 172.21.15.28:0/3085988034 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f803adf0 con 0x7fd504155640 2024-09-13T16:46:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.982+0000 7fd500c00640 1 -- 172.21.15.28:0/3085988034 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fd4f8046440 con 0x7fd504155640 2024-09-13T16:46:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.982+0000 7fd500c00640 1 -- 172.21.15.28:0/3085988034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504155640 msgr2=0x7fd50414df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.982+0000 7fd500c00640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504155640 0x7fd50414df00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd5040c4fe0 tx=0x7fd4f80046c0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.982+0000 7fd500c00640 1 -- 172.21.15.28:0/3085988034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504155640 msgr2=0x7fd50414df00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:46:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.982+0000 7fd500c00640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504155640 0x7fd50414df00 secure :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd5040c4fe0 tx=0x7fd4f80046c0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.990+0000 7fd50a000640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504153490 0x7fd5041538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.990+0000 7fd50a000640 1 -- 172.21.15.28:0/3085988034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5040c8260 con 0x7fd504153490 2024-09-13T16:46:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.991+0000 7fd508c00640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504153490 0x7fd5041538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.991+0000 7fd508c00640 1 -- 172.21.15.28:0/3085988034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd5040c5fd0 con 0x7fd504153490 2024-09-13T16:46:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.992+0000 7fd508c00640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504153490 0x7fd5041538a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd4f80097a0 tx=0x7fd4f8008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.992+0000 7fd500c00640 1 -- 172.21.15.28:0/3085988034 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8003bb0 con 0x7fd504153490 2024-09-13T16:46:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.992+0000 7fd500c00640 1 -- 172.21.15.28:0/3085988034 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+5101 (secure 0 0 0) 0x7fd4f804f070 con 0x7fd504153490 2024-09-13T16:46:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.992+0000 7fd500c00640 1 -- 172.21.15.28:0/3085988034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504153490 msgr2=0x7fd5041538a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.992+0000 7fd500c00640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504153490 0x7fd5041538a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd4f80097a0 tx=0x7fd4f8008030 comp rx=0 tx=0).stop 2024-09-13T16:46:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.993+0000 7fd500c00640 1 -- 172.21.15.28:0/3085988034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504153490 msgr2=0x7fd5041538a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:46:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.993+0000 7fd500c00640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504153490 0x7fd5041538a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:22.995 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:46:22.995 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:46:22.995 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:46:22.995 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "leader", 2024-09-13T16:46:22.995 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 16, 2024-09-13T16:46:22.995 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:46:22.995 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:46:22.995 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:46:22.995 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:46:22.996 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:22.996 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 12, 2024-09-13T16:46:22.996 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:46:22.996 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:46:22.996 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:46:22.996 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:22.996 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:22.996 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:22.996 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:22.996 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:22.996 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:22.996 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:22.996 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:22.997 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:22.997 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:22.997 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:22.997 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:46:22.997 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:46:22.997 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:22.997 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:22.997 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:22.997 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:22.997 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:22.997 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:22.997 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:22.997 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:22.998 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:22.998 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:22.998 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:22.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:22.998 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:46:22.998 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:46:22.998 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:46:22.998 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:46:22.998 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:46:22.998 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:46:22.998 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:46:22.998 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:46:22.998 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:46:22.999 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:46:22.999 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:46:22.999 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:46:22.999 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:46:22.999 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:46:22.999 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:46:22.999 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:46:22.999 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:46:22.999 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:22.999 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:22.999 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:22.999 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:22.999 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:22.999 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:23.000 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:23.000 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:23.000 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:23.000 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:23.000 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:23.000 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:46:23.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.000 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:46:23.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.000 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:46:23.000 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:46:23.000 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:23.000 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:46:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:46:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:46:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:46:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:46:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:46:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:23.003 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:46:23.003 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:23.003 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:46:23.003 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.003 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.003 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:23.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.003 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:46:23.003 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:46:23.003 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:23.004 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:23.004 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:23.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.004 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:46:23.004 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:46:23.004 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:23.004 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:23.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:23.004 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:46:23.004 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:23.005 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:46:23.005 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.005 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.005 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:23.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.005 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:46:23.005 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:46:23.005 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:23.005 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:23.005 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:23.005 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.005 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:23.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.006 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:46:23.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:46:23.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.006 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:23.006 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:23.006 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:46:23.006 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.006 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:23.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:46:23.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.006 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:23.006 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:23.006 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:46:23.006 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.006 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:23.007 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:46:23.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.007 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:23.007 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:23.007 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 5 2024-09-13T16:46:23.007 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.007 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:23.007 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:46:23.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.007 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x2f018fb87aa4aafe", 2024-09-13T16:46:23.007 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "luminous", 2024-09-13T16:46:23.007 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:46:23.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4 2024-09-13T16:46:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": [ 2024-09-13T16:46:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:46:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:23.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.009 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:46:23.009 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:46:23.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.993+0000 7fd50a000640 1 -- 172.21.15.28:0/3085988034 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd4e406ac00 msgr2=0x7fd4e406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.993+0000 7fd50a000640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd4e406ac00 0x7fd4e406d0c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd5040c4070 tx=0x7fd4fc013a20 comp rx=0 tx=0).stop 2024-09-13T16:46:23.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.993+0000 7fd50a000640 1 -- 172.21.15.28:0/3085988034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5040ce5e0 msgr2=0x7fd5041ee340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.993+0000 7fd50a000640 1 --2- 172.21.15.28:0/3085988034 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5040ce5e0 0x7fd5041ee340 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000c480 tx=0x7fd4f000c940 comp rx=0 tx=0).stop 2024-09-13T16:46:23.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.993+0000 7fd509600640 1 -- 172.21.15.28:0/3085988034 reap_dead start 2024-09-13T16:46:23.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.993+0000 7fd509600640 1 -- 172.21.15.28:0/3085988034 reap_dead start 2024-09-13T16:46:23.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.993+0000 7fd50a000640 1 -- 172.21.15.28:0/3085988034 shutdown_connections 2024-09-13T16:46:23.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.993+0000 7fd50a000640 1 -- 172.21.15.28:0/3085988034 >> 172.21.15.28:0/3085988034 conn(0x7fd50406c880 msgr2=0x7fd5040cd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:23.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.993+0000 7fd50a000640 1 -- 172.21.15.28:0/3085988034 shutdown_connections 2024-09-13T16:46:23.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:22.993+0000 7fd50a000640 1 -- 172.21.15.28:0/3085988034 wait complete. 2024-09-13T16:46:23.011 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:46:23.011 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-13T16:46:23.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T16:46:23.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.329+0000 7fe399600640 1 Processor -- start 2024-09-13T16:46:23.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.329+0000 7fe399600640 1 -- start start 2024-09-13T16:46:23.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.335+0000 7fe399600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe3941574f0 0x7fe39406d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.335+0000 7fe399600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe39406ddc0 0x7fe3940c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.335+0000 7fe399600640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39415dee0 con 0x7fe394157110 2024-09-13T16:46:23.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.335+0000 7fe399600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe394071c60 con 0x7fe3941574f0 2024-09-13T16:46:23.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.335+0000 7fe399600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3940c1ff0 con 0x7fe39406ddc0 2024-09-13T16:46:23.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.335+0000 7fe392a00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fe394157110 0x7fe3940c5580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:45520/0 (socket says 172.21.15.28:45520) 2024-09-13T16:46:23.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.335+0000 7fe392a00640 1 -- 172.21.15.28:0/654544019 learned_addr learned my addr 172.21.15.28:0/654544019 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:46:23.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.336+0000 7fe393400640 1 --2- 172.21.15.28:0/654544019 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe39406ddc0 0x7fe3940c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.336+0000 7fe392000640 1 -- 172.21.15.28:0/654544019 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886687199 0 0) 0x7fe39415dee0 con 0x7fe394157110 2024-09-13T16:46:23.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.336+0000 7fe392000640 1 -- 172.21.15.28:0/654544019 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe378003520 con 0x7fe394157110 2024-09-13T16:46:23.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.336+0000 7fe392000640 1 -- 172.21.15.28:0/654544019 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 64463319 0 0) 0x7fe378003520 con 0x7fe394157110 2024-09-13T16:46:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.336+0000 7fe392000640 1 -- 172.21.15.28:0/654544019 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe39406ddc0 msgr2=0x7fe3940c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:46:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.336+0000 7fe392000640 1 --2- 172.21.15.28:0/654544019 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe39406ddc0 0x7fe3940c6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.336+0000 7fe392000640 1 -- 172.21.15.28:0/654544019 >> v2:172.21.15.28:3300/0 conn(0x7fe3941574f0 msgr2=0x7fe39406d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:46:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.336+0000 7fe392000640 1 --2- 172.21.15.28:0/654544019 >> v2:172.21.15.28:3300/0 conn(0x7fe3941574f0 0x7fe39406d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.336+0000 7fe392000640 1 -- 172.21.15.28:0/654544019 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39406d3f0 con 0x7fe394157110 2024-09-13T16:46:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.336+0000 7fe392000640 1 -- 172.21.15.28:0/654544019 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1654382301 0 0) 0x7fe384001670 con 0x7fe394157110 2024-09-13T16:46:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.336+0000 7fe392000640 1 -- 172.21.15.28:0/654544019 >> v1:172.21.15.191:6789/0 conn(0x7fe394157110 legacy=0x7fe3940c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.336+0000 7fe392000640 1 --2- 172.21.15.28:0/654544019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe378003970 0x7fe378003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.336+0000 7fe392000640 1 --2- 172.21.15.28:0/654544019 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3780047c0 0x7fe378004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.337+0000 7fe392000640 1 --2- 172.21.15.28:0/654544019 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe378005180 0x7fe37800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.337+0000 7fe392000640 1 -- 172.21.15.28:0/654544019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe394071c60 con 0x7fe378003970 2024-09-13T16:46:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.337+0000 7fe392000640 1 -- 172.21.15.28:0/654544019 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3940c1ff0 con 0x7fe378005180 2024-09-13T16:46:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.337+0000 7fe392000640 1 -- 172.21.15.28:0/654544019 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe37800c070 con 0x7fe3780047c0 2024-09-13T16:46:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.338+0000 7fe38a000640 1 --2- 172.21.15.28:0/654544019 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3780047c0 0x7fe378004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.338+0000 7fe392a00640 1 --2- 172.21.15.28:0/654544019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe378003970 0x7fe378003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.338+0000 7fe38a000640 1 -- 172.21.15.28:0/654544019 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe378005180 msgr2=0x7fe37800b980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-13T16:46:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.338+0000 7fe38a000640 1 --2- 172.21.15.28:0/654544019 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe378005180 0x7fe37800b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.338+0000 7fe38a000640 1 -- 172.21.15.28:0/654544019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe378003970 msgr2=0x7fe378003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:46:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.338+0000 7fe38a000640 1 --2- 172.21.15.28:0/654544019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe378003970 0x7fe378003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.338+0000 7fe38a000640 1 -- 172.21.15.28:0/654544019 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39406d3f0 con 0x7fe3780047c0 2024-09-13T16:46:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.340+0000 7fe38a000640 1 --2- 172.21.15.28:0/654544019 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3780047c0 0x7fe378004c40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe37c009860 tx=0x7fe3780057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7aeac75dac0b3a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.341+0000 7fe392000640 1 -- 172.21.15.28:0/654544019 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c02aa80 con 0x7fe3780047c0 2024-09-13T16:46:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.341+0000 7fe392000640 1 -- 172.21.15.28:0/654544019 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe37c02ac00 con 0x7fe3780047c0 2024-09-13T16:46:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.341+0000 7fe392000640 1 -- 172.21.15.28:0/654544019 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c032440 con 0x7fe3780047c0 2024-09-13T16:46:23.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.350+0000 7fe399600640 1 -- 172.21.15.28:0/654544019 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3780047c0 msgr2=0x7fe378004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.350+0000 7fe399600640 1 --2- 172.21.15.28:0/654544019 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3780047c0 0x7fe378004c40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe37c009860 tx=0x7fe3780057f0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.350+0000 7fe393400640 1 -- 172.21.15.28:0/654544019 reap_dead start 2024-09-13T16:46:23.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.350+0000 7fe399600640 1 -- 172.21.15.28:0/654544019 shutdown_connections 2024-09-13T16:46:23.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.350+0000 7fe399600640 1 -- 172.21.15.28:0/654544019 >> 172.21.15.28:0/654544019 conn(0x7fe3940ca6b0 msgr2=0x7fe3940caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:23.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.352+0000 7fe399600640 1 -- 172.21.15.28:0/654544019 shutdown_connections 2024-09-13T16:46:23.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.352+0000 7fe399600640 1 -- 172.21.15.28:0/654544019 wait complete. 2024-09-13T16:46:23.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.358+0000 7fe399600640 1 Processor -- start 2024-09-13T16:46:23.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.358+0000 7fe399600640 1 -- start start 2024-09-13T16:46:23.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.359+0000 7fe399600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39414d940 0x7fe3941598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.359+0000 7fe399600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe394159e00 0x7fe39414fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.359+0000 7fe399600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe394150040 0x7fe3941f5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.359+0000 7fe399600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3940c7a20 con 0x7fe394150040 2024-09-13T16:46:23.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.359+0000 7fe399600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe394157940 con 0x7fe39414d940 2024-09-13T16:46:23.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.359+0000 7fe399600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39415a260 con 0x7fe394159e00 2024-09-13T16:46:23.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.360+0000 7fe38a000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe394159e00 0x7fe39414fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.360+0000 7fe38a000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe394159e00 0x7fe39414fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:39736/0 (socket says 172.21.15.28:39736) 2024-09-13T16:46:23.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.360+0000 7fe38a000640 1 -- 172.21.15.28:0/1217363498 learned_addr learned my addr 172.21.15.28:0/1217363498 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:23.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.360+0000 7fe392a00640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39414d940 0x7fe3941598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.360+0000 7fe38a000640 1 -- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39414d940 msgr2=0x7fe3941598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.360+0000 7fe38a000640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39414d940 0x7fe3941598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.360+0000 7fe38a000640 1 -- 172.21.15.28:0/1217363498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe394150040 msgr2=0x7fe3941f5b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:46:23.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.361+0000 7fe393400640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe394150040 0x7fe3941f5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.361+0000 7fe38a000640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe394150040 0x7fe3941f5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.361+0000 7fe38a000640 1 -- 172.21.15.28:0/1217363498 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3941582d0 con 0x7fe394159e00 2024-09-13T16:46:23.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.361+0000 7fe392a00640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39414d940 0x7fe3941598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:46:23.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.361+0000 7fe393400640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe394150040 0x7fe3941f5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:46:23.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.362+0000 7fe38a000640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe394159e00 0x7fe39414fb00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe37c009980 tx=0x7fe37c0047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:23.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.363+0000 7fe38b400640 1 -- 172.21.15.28:0/1217363498 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c003db0 con 0x7fe394159e00 2024-09-13T16:46:23.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.363+0000 7fe38b400640 1 -- 172.21.15.28:0/1217363498 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe37c0044c0 con 0x7fe394159e00 2024-09-13T16:46:23.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.363+0000 7fe38b400640 1 -- 172.21.15.28:0/1217363498 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c032680 con 0x7fe394159e00 2024-09-13T16:46:23.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.363+0000 7fe399600640 1 -- 172.21.15.28:0/1217363498 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe394150800 con 0x7fe394159e00 2024-09-13T16:46:23.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.364+0000 7fe399600640 1 -- 172.21.15.28:0/1217363498 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3941504c0 con 0x7fe394159e00 2024-09-13T16:46:23.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.366+0000 7fe399600640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3941f6060 0x7fe3941f6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.366+0000 7fe399600640 1 -- 172.21.15.28:0/1217363498 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39415a260 con 0x7fe3941f6060 2024-09-13T16:46:23.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.366+0000 7fe392a00640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3941f6060 0x7fe3941f6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.366+0000 7fe38b400640 1 -- 172.21.15.28:0/1217363498 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe37c039050 con 0x7fe394159e00 2024-09-13T16:46:23.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.367+0000 7fe38b400640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe37006ac80 0x7fe37006d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.367+0000 7fe38b400640 1 -- 172.21.15.28:0/1217363498 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe37c0a9090 con 0x7fe394159e00 2024-09-13T16:46:23.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.368+0000 7fe393400640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe37006ac80 0x7fe37006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.368+0000 7fe392a00640 1 -- 172.21.15.28:0/1217363498 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe394150680 con 0x7fe3941f6060 2024-09-13T16:46:23.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.369+0000 7fe392a00640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3941f6060 0x7fe3941f6440 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe38400eb00 tx=0x7fe384001db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:23.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.369+0000 7fe393400640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe37006ac80 0x7fe37006d140 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe39406aed0 tx=0x7fe38c00b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.371+0000 7fe38b400640 1 -- 172.21.15.28:0/1217363498 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe384004cb0 con 0x7fe3941f6060 2024-09-13T16:46:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.371+0000 7fe38b400640 1 -- 172.21.15.28:0/1217363498 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fe38400d040 con 0x7fe3941f6060 2024-09-13T16:46:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.371+0000 7fe38b400640 1 -- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3941f6060 msgr2=0x7fe3941f6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.371+0000 7fe38b400640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3941f6060 0x7fe3941f6440 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe38400eb00 tx=0x7fe384001db0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.371+0000 7fe38b400640 1 -- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3941f6060 msgr2=0x7fe3941f6440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:46:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.371+0000 7fe38b400640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3941f6060 0x7fe3941f6440 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe38400eb00 tx=0x7fe384001db0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.381+0000 7fe399600640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39415abc0 0x7fe39415afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.381+0000 7fe399600640 1 -- 172.21.15.28:0/1217363498 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe394066c20 con 0x7fe39415abc0 2024-09-13T16:46:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.382+0000 7fe392a00640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39415abc0 0x7fe39415afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.382+0000 7fe392a00640 1 -- 172.21.15.28:0/1217363498 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe394159220 con 0x7fe39415abc0 2024-09-13T16:46:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.383+0000 7fe392a00640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39415abc0 0x7fe39415afd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe38400eb00 tx=0x7fe384001930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:23.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.383+0000 7fe38b400640 1 -- 172.21.15.28:0/1217363498 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe384022070 con 0x7fe39415abc0 2024-09-13T16:46:23.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.383+0000 7fe38b400640 1 -- 172.21.15.28:0/1217363498 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4458 (secure 0 0 0) 0x7fe384004e10 con 0x7fe39415abc0 2024-09-13T16:46:23.386 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:46:23.386 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:46:23.386 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:46:23.386 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:46:23.386 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 16, 2024-09-13T16:46:23.386 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:46:23.386 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:46:23.386 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:46:23.386 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:46:23.387 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:23.387 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 12, 2024-09-13T16:46:23.387 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:46:23.387 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:46:23.387 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:46:23.387 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:23.387 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:23.387 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:23.387 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:23.387 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:23.387 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:23.387 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:23.387 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:23.388 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:23.388 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:23.388 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:23.388 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:46:23.388 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:46:23.388 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:23.388 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:23.388 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:23.388 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:23.388 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:23.388 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:23.388 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:23.389 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:23.389 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:23.389 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:23.389 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:23.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.389 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:46:23.389 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:46:23.389 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:46:23.389 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:46:23.389 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:46:23.389 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:46:23.389 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:46:23.389 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:46:23.390 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:46:23.390 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:46:23.390 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:46:23.390 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:46:23.390 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:46:23.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:46:23.390 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:46:23.390 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:46:23.390 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:46:23.390 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:23.390 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:23.390 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:23.391 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:23.391 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:23.391 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:23.391 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:23.391 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:23.391 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:23.391 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:23.391 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:23.391 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:46:23.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.391 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:46:23.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.391 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:46:23.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:46:23.391 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:46:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:46:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:46:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:46:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:46:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:46:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:46:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:46:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:46:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:46:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:23.394 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:46:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:46:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:46:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:23.395 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:46:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:46:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:46:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:46:23.396 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:46:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:46:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:46:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:46:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.397 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:23.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.398 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:46:23.398 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:46:23.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.384+0000 7fe38b400640 1 -- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39415abc0 msgr2=0x7fe39415afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.384+0000 7fe38b400640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39415abc0 0x7fe39415afd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe38400eb00 tx=0x7fe384001930 comp rx=0 tx=0).stop 2024-09-13T16:46:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.384+0000 7fe38b400640 1 -- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39415abc0 msgr2=0x7fe39415afd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:46:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.384+0000 7fe38b400640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39415abc0 0x7fe39415afd0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.384+0000 7fe399600640 1 -- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe37006ac80 msgr2=0x7fe37006d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.384+0000 7fe399600640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe37006ac80 0x7fe37006d140 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe39406aed0 tx=0x7fe38c00b040 comp rx=0 tx=0).stop 2024-09-13T16:46:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.385+0000 7fe393400640 1 -- 172.21.15.28:0/1217363498 reap_dead start 2024-09-13T16:46:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.385+0000 7fe399600640 1 -- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe394159e00 msgr2=0x7fe39414fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.385+0000 7fe399600640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe394159e00 0x7fe39414fb00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe37c009980 tx=0x7fe37c0047d0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.385+0000 7fe399600640 1 -- 172.21.15.28:0/1217363498 shutdown_connections 2024-09-13T16:46:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.385+0000 7fe399600640 1 --2- 172.21.15.28:0/1217363498 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe394159e00 0x7fe39414fb00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.385+0000 7fe399600640 1 -- 172.21.15.28:0/1217363498 >> 172.21.15.28:0/1217363498 conn(0x7fe3940ca6b0 msgr2=0x7fe3940c2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.385+0000 7fe399600640 1 -- 172.21.15.28:0/1217363498 shutdown_connections 2024-09-13T16:46:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.385+0000 7fe399600640 1 -- 172.21.15.28:0/1217363498 wait complete. 2024-09-13T16:46:23.407 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:46:23.407 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-13T16:46:23.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T16:46:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.618+0000 7fd157e00640 1 Processor -- start 2024-09-13T16:46:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.618+0000 7fd157e00640 1 -- start start 2024-09-13T16:46:23.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.619+0000 7fd157e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd1580ce5e0 0x7fd1581595a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.619+0000 7fd157e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd1580ce9c0 0x7fd15815b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.619+0000 7fd157e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd158159f90 con 0x7fd1580c94d0 2024-09-13T16:46:23.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.619+0000 7fd157e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd158159ae0 con 0x7fd1580ce9c0 2024-09-13T16:46:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.619+0000 7fd157e00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1580c6900 con 0x7fd1580ce5e0 2024-09-13T16:46:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.620+0000 7fd156a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd1580ce5e0 0x7fd1581595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.620+0000 7fd156a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd1580ce5e0 0x7fd1581595a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:48016/0 (socket says 172.21.15.28:48016) 2024-09-13T16:46:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.620+0000 7fd156a00640 1 -- 172.21.15.28:0/176648417 learned_addr learned my addr 172.21.15.28:0/176648417 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.620+0000 7fd156000640 1 --2- 172.21.15.28:0/176648417 >> v2:172.21.15.28:3300/0 conn(0x7fd1580ce9c0 0x7fd15815b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.620+0000 7fd156a00640 1 -- 172.21.15.28:0/176648417 >> v2:172.21.15.28:3300/0 conn(0x7fd1580ce9c0 msgr2=0x7fd15815b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:46:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.620+0000 7fd156a00640 1 --2- 172.21.15.28:0/176648417 >> v2:172.21.15.28:3300/0 conn(0x7fd1580ce9c0 0x7fd15815b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.620+0000 7fd156a00640 1 -- 172.21.15.28:0/176648417 >> v1:172.21.15.28:6789/0 conn(0x7fd1580c94d0 legacy=0x7fd1580c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:46:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.621+0000 7fd156a00640 1 -- 172.21.15.28:0/176648417 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1580c5800 con 0x7fd1580ce5e0 2024-09-13T16:46:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.621+0000 7fd156000640 1 --2- 172.21.15.28:0/176648417 >> v2:172.21.15.28:3300/0 conn(0x7fd1580ce9c0 0x7fd15815b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:46:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.622+0000 7fd156a00640 1 --2- 172.21.15.28:0/176648417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd1580ce5e0 0x7fd1581595a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd1580c8fc0 tx=0x7fd14802a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d68af0d41d6985d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.622+0000 7fd155600640 1 -- 172.21.15.28:0/176648417 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148032640 con 0x7fd1580ce5e0 2024-09-13T16:46:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.622+0000 7fd155600640 1 -- 172.21.15.28:0/176648417 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1480327a0 con 0x7fd1580ce5e0 2024-09-13T16:46:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.622+0000 7fd155600640 1 -- 172.21.15.28:0/176648417 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148032b70 con 0x7fd1580ce5e0 2024-09-13T16:46:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.623+0000 7fd157e00640 1 -- 172.21.15.28:0/176648417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd1580ce5e0 msgr2=0x7fd1581595a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.623+0000 7fd157e00640 1 --2- 172.21.15.28:0/176648417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd1580ce5e0 0x7fd1581595a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd1580c8fc0 tx=0x7fd14802a330 comp rx=0 tx=0).stop 2024-09-13T16:46:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.623+0000 7fd157e00640 1 -- 172.21.15.28:0/176648417 shutdown_connections 2024-09-13T16:46:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.623+0000 7fd157e00640 1 --2- 172.21.15.28:0/176648417 >> v2:172.21.15.28:3300/0 conn(0x7fd1580ce9c0 0x7fd15815b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.623+0000 7fd157e00640 1 --2- 172.21.15.28:0/176648417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd1580ce5e0 0x7fd1581595a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.623+0000 7fd157e00640 1 -- 172.21.15.28:0/176648417 >> 172.21.15.28:0/176648417 conn(0x7fd15806c880 msgr2=0x7fd1580715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.623+0000 7fd157e00640 1 -- 172.21.15.28:0/176648417 shutdown_connections 2024-09-13T16:46:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.623+0000 7fd157e00640 1 -- 172.21.15.28:0/176648417 wait complete. 2024-09-13T16:46:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.631+0000 7fd157e00640 1 Processor -- start 2024-09-13T16:46:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.632+0000 7fd157e00640 1 -- start start 2024-09-13T16:46:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.633+0000 7fd157e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd1580c94d0 0x7fd158155c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.633+0000 7fd157e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1580ce5e0 0x7fd1580c34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.633+0000 7fd157e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1580ce9c0 0x7fd158153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.633+0000 7fd157e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd158156150 con 0x7fd1580ce5e0 2024-09-13T16:46:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.633+0000 7fd157e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15814fc60 con 0x7fd1580c94d0 2024-09-13T16:46:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.633+0000 7fd157e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1581edee0 con 0x7fd1580ce9c0 2024-09-13T16:46:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.633+0000 7fd156000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1580ce5e0 0x7fd1580c34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.633+0000 7fd156000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1580ce5e0 0x7fd1580c34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51000/0 (socket says 172.21.15.28:51000) 2024-09-13T16:46:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.633+0000 7fd156000640 1 -- 172.21.15.28:0/3192179401 learned_addr learned my addr 172.21.15.28:0/3192179401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.634+0000 7fd157400640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1580ce9c0 0x7fd158153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.634+0000 7fd156a00640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd1580c94d0 0x7fd158155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.634+0000 7fd156000640 1 -- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1580ce9c0 msgr2=0x7fd158153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.634+0000 7fd156000640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1580ce9c0 0x7fd158153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.634+0000 7fd156000640 1 -- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd1580c94d0 msgr2=0x7fd158155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.634+0000 7fd156000640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd1580c94d0 0x7fd158155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.634+0000 7fd156000640 1 -- 172.21.15.28:0/3192179401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd144009400 con 0x7fd1580ce5e0 2024-09-13T16:46:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.634+0000 7fd156a00640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd1580c94d0 0x7fd158155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:46:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.634+0000 7fd156000640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1580ce5e0 0x7fd1580c34f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd14400e960 tx=0x7fd14400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.635+0000 7fd14ea00640 1 -- 172.21.15.28:0/3192179401 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14400ca00 con 0x7fd1580ce5e0 2024-09-13T16:46:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.635+0000 7fd14ea00640 1 -- 172.21.15.28:0/3192179401 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14400cb60 con 0x7fd1580ce5e0 2024-09-13T16:46:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.635+0000 7fd14ea00640 1 -- 172.21.15.28:0/3192179401 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd144013470 con 0x7fd1580ce5e0 2024-09-13T16:46:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.635+0000 7fd157e00640 1 -- 172.21.15.28:0/3192179401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1480094d0 con 0x7fd1580ce5e0 2024-09-13T16:46:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.635+0000 7fd157e00640 1 -- 172.21.15.28:0/3192179401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1581ee4a0 con 0x7fd1580ce5e0 2024-09-13T16:46:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.636+0000 7fd157e00640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1581ee6c0 0x7fd158154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.636+0000 7fd157e00640 1 -- 172.21.15.28:0/3192179401 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd158072160 con 0x7fd1581ee6c0 2024-09-13T16:46:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.637+0000 7fd156a00640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1581ee6c0 0x7fd158154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.637+0000 7fd156a00640 1 -- 172.21.15.28:0/3192179401 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1580c5cf0 con 0x7fd1581ee6c0 2024-09-13T16:46:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.637+0000 7fd156a00640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1581ee6c0 0x7fd158154920 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd15806c200 tx=0x7fd1480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.637+0000 7fd14ea00640 1 -- 172.21.15.28:0/3192179401 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd1440135d0 con 0x7fd1580ce5e0 2024-09-13T16:46:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.638+0000 7fd14ea00640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd13806ac00 0x7fd13806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.638+0000 7fd14ea00640 1 -- 172.21.15.28:0/3192179401 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd144089950 con 0x7fd1580ce5e0 2024-09-13T16:46:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.638+0000 7fd157400640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd13806ac00 0x7fd13806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.639+0000 7fd14ea00640 1 -- 172.21.15.28:0/3192179401 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14803c940 con 0x7fd1581ee6c0 2024-09-13T16:46:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.639+0000 7fd14ea00640 1 -- 172.21.15.28:0/3192179401 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fd14803caa0 con 0x7fd1581ee6c0 2024-09-13T16:46:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.639+0000 7fd14ea00640 1 -- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1581ee6c0 msgr2=0x7fd158154920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.639+0000 7fd14ea00640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1581ee6c0 0x7fd158154920 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd15806c200 tx=0x7fd1480046c0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.639+0000 7fd14ea00640 1 -- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1581ee6c0 msgr2=0x7fd158154920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:46:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.639+0000 7fd14ea00640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1581ee6c0 0x7fd158154920 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.639+0000 7fd157400640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd13806ac00 0x7fd13806d0c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd1500012e0 tx=0x7fd150009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.645+0000 7fd157e00640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1580cdc30 0x7fd158154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.645+0000 7fd157e00640 1 -- 172.21.15.28:0/3192179401 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1580c8260 con 0x7fd1580cdc30 2024-09-13T16:46:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.646+0000 7fd156a00640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1580cdc30 0x7fd158154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.646+0000 7fd156a00640 1 -- 172.21.15.28:0/3192179401 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd1580c5fd0 con 0x7fd1580cdc30 2024-09-13T16:46:23.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.646+0000 7fd156a00640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1580cdc30 0x7fd158154e60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd148000c00 tx=0x7fd148009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:23.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.647+0000 7fd14ea00640 1 -- 172.21.15.28:0/3192179401 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148049a60 con 0x7fd1580cdc30 2024-09-13T16:46:23.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.647+0000 7fd14ea00640 1 -- 172.21.15.28:0/3192179401 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4790 (secure 0 0 0) 0x7fd14804d070 con 0x7fd1580cdc30 2024-09-13T16:46:23.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.647+0000 7fd14ea00640 1 -- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1580cdc30 msgr2=0x7fd158154e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.647+0000 7fd14ea00640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1580cdc30 0x7fd158154e60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd148000c00 tx=0x7fd148009240 comp rx=0 tx=0).stop 2024-09-13T16:46:23.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.647+0000 7fd14ea00640 1 -- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1580cdc30 msgr2=0x7fd158154e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:46:23.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.647+0000 7fd14ea00640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd1580cdc30 0x7fd158154e60 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd148000c00 tx=0x7fd148009240 comp rx=0 tx=0).stop 2024-09-13T16:46:23.650 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:46:23.650 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:46:23.650 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:46:23.650 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:46:23.650 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 16, 2024-09-13T16:46:23.651 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:46:23.651 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:46:23.651 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:46:23.651 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:46:23.651 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:23.651 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 12, 2024-09-13T16:46:23.652 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:46:23.652 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:46:23.652 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:46:23.652 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:23.652 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:23.652 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:23.653 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:23.653 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:23.653 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:23.653 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:23.653 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:23.653 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:23.654 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:23.654 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:23.654 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:46:23.654 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:46:23.654 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:23.654 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:23.655 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.656 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:46:23.657 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:46:23.657 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:46:23.657 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:46:23.657 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:46:23.657 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:46:23.657 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:46:23.658 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:46:23.658 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:46:23.658 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:46:23.658 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:46:23.658 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:46:23.659 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:46:23.659 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:46:23.659 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:46:23.659 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:46:23.659 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:46:23.659 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:23.660 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:23.660 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:23.660 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:23.660 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:23.660 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:23.660 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:23.661 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:23.661 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:23.661 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:23.661 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:23.661 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:46:23.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.662 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:46:23.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.662 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:46:23.662 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:46:23.662 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:23.663 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:23.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:23.663 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:46:23.663 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.664 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:23.664 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:46:23.664 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.664 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.664 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:23.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.665 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:46:23.665 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:46:23.665 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:23.665 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:23.665 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:23.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.666 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:46:23.666 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:46:23.666 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:23.666 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:23.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.667 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:23.667 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:46:23.667 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.667 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:23.667 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:46:23.667 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.667 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.667 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:23.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.668 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:46:23.668 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:46:23.668 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:23.668 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:23.668 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:23.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.668 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:46:23.668 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:46:23.668 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:23.668 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:23.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.669 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:23.669 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:46:23.669 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.669 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:23.669 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:46:23.669 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:23.669 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.669 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:23.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.669 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:46:23.670 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:46:23.670 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:23.670 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:23.670 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:23.670 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.670 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:23.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.670 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:46:23.670 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:46:23.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.670 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:23.670 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:23.671 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:46:23.671 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.671 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:23.671 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:46:23.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.671 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:23.671 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:23.671 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:46:23.671 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.671 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:23.671 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:46:23.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.671 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:23.672 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:23.672 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:46:23.672 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.672 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:23.672 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:46:23.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:23.672 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:23.672 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:23.672 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:46:23.672 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:23.672 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:23.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:23.673 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:46:23.673 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:46:23.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.648+0000 7fd157e00640 1 -- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd13806ac00 msgr2=0x7fd13806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.648+0000 7fd157e00640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd13806ac00 0x7fd13806d0c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd1500012e0 tx=0x7fd150009040 comp rx=0 tx=0).stop 2024-09-13T16:46:23.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.648+0000 7fd157e00640 1 -- 172.21.15.28:0/3192179401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1580ce5e0 msgr2=0x7fd1580c34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.648+0000 7fd157e00640 1 --2- 172.21.15.28:0/3192179401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1580ce5e0 0x7fd1580c34f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd14400e960 tx=0x7fd14400ee20 comp rx=0 tx=0).stop 2024-09-13T16:46:23.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.648+0000 7fd157400640 1 -- 172.21.15.28:0/3192179401 reap_dead start 2024-09-13T16:46:23.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.648+0000 7fd157400640 1 -- 172.21.15.28:0/3192179401 reap_dead start 2024-09-13T16:46:23.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.648+0000 7fd157e00640 1 -- 172.21.15.28:0/3192179401 shutdown_connections 2024-09-13T16:46:23.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.648+0000 7fd157e00640 1 -- 172.21.15.28:0/3192179401 >> 172.21.15.28:0/3192179401 conn(0x7fd15806c880 msgr2=0x7fd158065c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:23.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.648+0000 7fd157e00640 1 -- 172.21.15.28:0/3192179401 shutdown_connections 2024-09-13T16:46:23.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.648+0000 7fd157e00640 1 -- 172.21.15.28:0/3192179401 wait complete. 2024-09-13T16:46:23.675 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:46:23.675 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-13T16:46:23.675 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-13T16:46:23.675 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-13T16:46:23.675 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-13T16:46:23.676 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-13T16:46:23.676 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:46:23.717 INFO:tasks.ceph.mon.a:Stopped 2024-09-13T16:46:23.718 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-13T16:46:23.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:46:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.925+0000 7f6448a00640 1 Processor -- start 2024-09-13T16:46:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.925+0000 7f6448a00640 1 -- start start 2024-09-13T16:46:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.926+0000 7f6448a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64440c5440 0x7f64440c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.926+0000 7f6448a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6444156610 0x7f6444150400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.926+0000 7f6448a00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64440c5df0 con 0x7f64440c5060 2024-09-13T16:46:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.926+0000 7f6448a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64440c2ab0 con 0x7f6444156610 2024-09-13T16:46:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.926+0000 7f6448a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64440c2d70 con 0x7f64440c5440 2024-09-13T16:46:23.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.926+0000 7f6443e00640 1 -- >> v2:172.21.15.28:3300/0 conn(0x7f6444156610 msgr2=0x7f6444150400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:46:23.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.927+0000 7f6443e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6444156610 0x7f6444150400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:23.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.927+0000 7f6442a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64440c5440 0x7f64440c44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.927+0000 7f6443400640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f64440c5060 0x7f64440c2300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:45522/0 (socket says 172.21.15.28:45522) 2024-09-13T16:46:23.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.927+0000 7f6442a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64440c5440 0x7f64440c44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:39858/0 (socket says 172.21.15.28:39858) 2024-09-13T16:46:23.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.927+0000 7f6443400640 1 -- 172.21.15.28:0/1151296149 learned_addr learned my addr 172.21.15.28:0/1151296149 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:46:23.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.927+0000 7f6442a00640 1 -- 172.21.15.28:0/1151296149 >> v2:172.21.15.28:3300/0 conn(0x7f6444156610 msgr2=0x7f6444150400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:46:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.927+0000 7f6442000640 1 -- 172.21.15.28:0/1151296149 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751189113 0 0) 0x7f64440c5df0 con 0x7f64440c5060 2024-09-13T16:46:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.927+0000 7f6442a00640 1 --2- 172.21.15.28:0/1151296149 >> v2:172.21.15.28:3300/0 conn(0x7f6444156610 0x7f6444150400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.927+0000 7f6442a00640 1 -- 172.21.15.28:0/1151296149 >> v1:172.21.15.191:6789/0 conn(0x7f64440c5060 legacy=0x7f64440c2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.927+0000 7f6442a00640 1 -- 172.21.15.28:0/1151296149 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64440c6040 con 0x7f64440c5440 2024-09-13T16:46:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.929+0000 7f6442a00640 1 --2- 172.21.15.28:0/1151296149 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64440c5440 0x7f64440c44d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f64300097a0 tx=0x7f643002a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afe58fd8d0fa6158 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.929+0000 7f6442000640 1 -- 172.21.15.28:0/1151296149 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6430032440 con 0x7f64440c5440 2024-09-13T16:46:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.929+0000 7f6442000640 1 -- 172.21.15.28:0/1151296149 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64300325a0 con 0x7f64440c5440 2024-09-13T16:46:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.929+0000 7f6442000640 1 -- 172.21.15.28:0/1151296149 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6430032930 con 0x7f64440c5440 2024-09-13T16:46:23.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.930+0000 7f6448a00640 1 -- 172.21.15.28:0/1151296149 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64440c5440 msgr2=0x7f64440c44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.930+0000 7f6448a00640 1 --2- 172.21.15.28:0/1151296149 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64440c5440 0x7f64440c44d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f64300097a0 tx=0x7f643002a230 comp rx=0 tx=0).stop 2024-09-13T16:46:23.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.933+0000 7f6448a00640 1 -- 172.21.15.28:0/1151296149 shutdown_connections 2024-09-13T16:46:23.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.933+0000 7f6448a00640 1 --2- 172.21.15.28:0/1151296149 >> v2:172.21.15.28:3300/0 conn(0x7f6444156610 0x7f6444150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.933+0000 7f6448a00640 1 --2- 172.21.15.28:0/1151296149 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64440c5440 0x7f64440c44d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.933+0000 7f6448a00640 1 -- 172.21.15.28:0/1151296149 >> 172.21.15.28:0/1151296149 conn(0x7f6444069e80 msgr2=0x7f644406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:23.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.938+0000 7f6448a00640 1 -- 172.21.15.28:0/1151296149 shutdown_connections 2024-09-13T16:46:23.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.938+0000 7f6448a00640 1 -- 172.21.15.28:0/1151296149 wait complete. 2024-09-13T16:46:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.940+0000 7f6448a00640 1 Processor -- start 2024-09-13T16:46:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.941+0000 7f6448a00640 1 -- start start 2024-09-13T16:46:23.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.942+0000 7f6448a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64440c5060 0x7f6444148b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.942+0000 7f6448a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64440c5440 0x7f64440700d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.942+0000 7f6442a00640 1 -- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64440c5440 msgr2=0x7f64440700d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:46:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.942+0000 7f6442a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64440c5440 0x7f64440700d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.942+0000 7f6448a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6444156610 0x7f64440c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.942+0000 7f6448a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644414fd00 con 0x7f64440c5440 2024-09-13T16:46:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.942+0000 7f6448a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64440c8420 con 0x7f64440c5060 2024-09-13T16:46:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.942+0000 7f6448a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64441f2060 con 0x7f6444156610 2024-09-13T16:46:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.942+0000 7f6443400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64440c5060 0x7f6444148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.942+0000 7f6443400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64440c5060 0x7f6444148b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:48106/0 (socket says 172.21.15.28:48106) 2024-09-13T16:46:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.942+0000 7f6443400640 1 -- 172.21.15.28:0/2353288472 learned_addr learned my addr 172.21.15.28:0/2353288472 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.943+0000 7f6443e00640 1 --2- 172.21.15.28:0/2353288472 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6444156610 0x7f64440c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.943+0000 7f6443400640 1 -- 172.21.15.28:0/2353288472 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6444156610 msgr2=0x7f64440c9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.943+0000 7f6443400640 1 --2- 172.21.15.28:0/2353288472 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6444156610 0x7f64440c9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.943+0000 7f6443400640 1 -- 172.21.15.28:0/2353288472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64440c5440 msgr2=0x7f64440700d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:46:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.943+0000 7f6443400640 1 --2- 172.21.15.28:0/2353288472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64440c5440 0x7f64440700d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.943+0000 7f6443400640 1 -- 172.21.15.28:0/2353288472 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6430009400 con 0x7f64440c5060 2024-09-13T16:46:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.943+0000 7f6443e00640 1 --2- 172.21.15.28:0/2353288472 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6444156610 0x7f64440c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:46:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.944+0000 7f6443400640 1 --2- 172.21.15.28:0/2353288472 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64440c5060 0x7f6444148b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f643400c500 tx=0x7f643400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.944+0000 7f643b400640 1 -- 172.21.15.28:0/2353288472 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6434007560 con 0x7f64440c5060 2024-09-13T16:46:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.945+0000 7f643b400640 1 -- 172.21.15.28:0/2353288472 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64340047e0 con 0x7f64440c5060 2024-09-13T16:46:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.945+0000 7f6448a00640 1 -- 172.21.15.28:0/2353288472 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6444143c80 con 0x7f64440c5060 2024-09-13T16:46:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.945+0000 7f643b400640 1 -- 172.21.15.28:0/2353288472 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6434004b90 con 0x7f64440c5060 2024-09-13T16:46:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.945+0000 7f6448a00640 1 -- 172.21.15.28:0/2353288472 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64441f1d80 con 0x7f64440c5060 2024-09-13T16:46:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.947+0000 7f6448a00640 1 -- 172.21.15.28:0/2353288472 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64440c7d00 con 0x7f64440c5060 2024-09-13T16:46:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.948+0000 7f643b400640 1 -- 172.21.15.28:0/2353288472 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6434004cf0 con 0x7f64440c5060 2024-09-13T16:46:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.949+0000 7f643b400640 1 --2- 172.21.15.28:0/2353288472 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f642406abe0 0x7f642406d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.950+0000 7f643b400640 1 -- 172.21.15.28:0/2353288472 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6434088830 con 0x7f64440c5060 2024-09-13T16:46:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.950+0000 7f6442a00640 1 --2- 172.21.15.28:0/2353288472 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f642406abe0 0x7f642406d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:23.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.951+0000 7f6442a00640 1 --2- 172.21.15.28:0/2353288472 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f642406abe0 0x7f642406d0a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6430034000 tx=0x7f643002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:23.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:23.954+0000 7f643b400640 1 -- 172.21.15.28:0/2353288472 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6434053980 con 0x7f64440c5060 2024-09-13T16:46:24.035 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:24.035 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 rewriters = 326294.66 kB/sec 2024-09-13T16:46:24.035 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 rewriters = 326215.19 kB/sec 2024-09-13T16:46:24.035 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 326294.66 kB/sec 2024-09-13T16:46:24.035 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 326294.66 kB/sec 2024-09-13T16:46:24.035 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 326294.66 kB/sec 2024-09-13T16:46:24.035 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 1048576.00 kB 2024-09-13T16:46:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:24.225+0000 7f6448a00640 1 -- 172.21.15.28:0/2353288472 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f64440c1af0 con 0x7f64440c5060 2024-09-13T16:46:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:24.226+0000 7f643b400640 1 -- 172.21.15.28:0/2353288472 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f6434018080 con 0x7f64440c5060 2024-09-13T16:46:24.228 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:46:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:24.229+0000 7f6448a00640 1 -- 172.21.15.28:0/2353288472 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f642406abe0 msgr2=0x7f642406d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:24.229+0000 7f6448a00640 1 --2- 172.21.15.28:0/2353288472 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f642406abe0 0x7f642406d0a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6430034000 tx=0x7f643002e000 comp rx=0 tx=0).stop 2024-09-13T16:46:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:24.229+0000 7f6448a00640 1 -- 172.21.15.28:0/2353288472 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64440c5060 msgr2=0x7f6444148b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:24.229+0000 7f6448a00640 1 --2- 172.21.15.28:0/2353288472 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64440c5060 0x7f6444148b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f643400c500 tx=0x7f643400c9c0 comp rx=0 tx=0).stop 2024-09-13T16:46:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:24.229+0000 7f6448a00640 1 -- 172.21.15.28:0/2353288472 shutdown_connections 2024-09-13T16:46:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:24.229+0000 7f6448a00640 1 --2- 172.21.15.28:0/2353288472 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f642406abe0 0x7f642406d0a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:24.229+0000 7f6448a00640 1 --2- 172.21.15.28:0/2353288472 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6444156610 0x7f64440c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:24.229+0000 7f6448a00640 1 --2- 172.21.15.28:0/2353288472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64440c5440 0x7f64440700d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:24.229+0000 7f6448a00640 1 --2- 172.21.15.28:0/2353288472 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64440c5060 0x7f6444148b20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:24.229+0000 7f6448a00640 1 -- 172.21.15.28:0/2353288472 >> 172.21.15.28:0/2353288472 conn(0x7f6444069e80 msgr2=0x7f6444156a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:24.229+0000 7f6448a00640 1 -- 172.21.15.28:0/2353288472 shutdown_connections 2024-09-13T16:46:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:24.229+0000 7f6448a00640 1 -- 172.21.15.28:0/2353288472 wait complete. 2024-09-13T16:46:26.703 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:26.703 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 readers = 4938308.00 kB/sec 2024-09-13T16:46:26.703 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 readers = 4918619.39 kB/sec 2024-09-13T16:46:26.703 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 4938308.00 kB/sec 2024-09-13T16:46:26.703 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 4938308.00 kB/sec 2024-09-13T16:46:26.703 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 4938308.00 kB/sec 2024-09-13T16:46:26.703 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 1048576.00 kB 2024-09-13T16:46:27.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:46:27.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.403+0000 7f52bce00640 1 Processor -- start 2024-09-13T16:46:27.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.403+0000 7f52bce00640 1 -- start start 2024-09-13T16:46:27.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.404+0000 7f52bce00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52b814fa80 0x7f52b80c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:27.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.405+0000 7f52bce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f52b80c6350 0x7f52b81559d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:27.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.405+0000 7f52bce00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b80c42b0 con 0x7f52b8072170 2024-09-13T16:46:27.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.405+0000 7f52bce00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b80c36e0 con 0x7f52b80c6350 2024-09-13T16:46:27.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.405+0000 7f52bce00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b80c3980 con 0x7f52b814fa80 2024-09-13T16:46:27.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.405+0000 7f52ad400640 1 -- >> v1:172.21.15.28:6789/0 conn(0x7f52b8072170 legacy=0x7f52b80c6ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.28:6789/0 2024-09-13T16:46:27.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.405+0000 7f52b6800640 1 -- >> v2:172.21.15.28:3300/0 conn(0x7f52b80c6350 msgr2=0x7f52b81559d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:46:27.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.405+0000 7f52b6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f52b80c6350 0x7f52b81559d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:27.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.405+0000 7f52b5e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52b814fa80 0x7f52b80c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:27.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.405+0000 7f52b5e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52b814fa80 0x7f52b80c1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:48122/0 (socket says 172.21.15.28:48122) 2024-09-13T16:46:27.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.405+0000 7f52b5e00640 1 -- 172.21.15.28:0/3187669408 learned_addr learned my addr 172.21.15.28:0/3187669408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:27.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.405+0000 7f52b5e00640 1 -- 172.21.15.28:0/3187669408 >> v2:172.21.15.28:3300/0 conn(0x7f52b80c6350 msgr2=0x7f52b81559d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:46:27.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.405+0000 7f52b5e00640 1 --2- 172.21.15.28:0/3187669408 >> v2:172.21.15.28:3300/0 conn(0x7f52b80c6350 0x7f52b81559d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:27.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.405+0000 7f52b5e00640 1 -- 172.21.15.28:0/3187669408 >> v1:172.21.15.28:6789/0 conn(0x7f52b8072170 legacy=0x7f52b80c6ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:46:27.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.405+0000 7f52b5e00640 1 -- 172.21.15.28:0/3187669408 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b80c3c60 con 0x7f52b814fa80 2024-09-13T16:46:27.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.406+0000 7f52b5e00640 1 --2- 172.21.15.28:0/3187669408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b814fa80 0x7f52b80c1960 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f52b806c9c0 tx=0x7f52a802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=326094bbfc07e951 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:27.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.407+0000 7f52b5400640 1 -- 172.21.15.28:0/3187669408 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a8032440 con 0x7f52b814fa80 2024-09-13T16:46:27.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.407+0000 7f52b5400640 1 -- 172.21.15.28:0/3187669408 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a80325a0 con 0x7f52b814fa80 2024-09-13T16:46:27.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.407+0000 7f52b5400640 1 -- 172.21.15.28:0/3187669408 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a8032930 con 0x7f52b814fa80 2024-09-13T16:46:27.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.407+0000 7f52bce00640 1 -- 172.21.15.28:0/3187669408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b814fa80 msgr2=0x7f52b80c1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:27.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.407+0000 7f52bce00640 1 --2- 172.21.15.28:0/3187669408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b814fa80 0x7f52b80c1960 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f52b806c9c0 tx=0x7f52a802a250 comp rx=0 tx=0).stop 2024-09-13T16:46:27.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.407+0000 7f52bce00640 1 -- 172.21.15.28:0/3187669408 shutdown_connections 2024-09-13T16:46:27.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.407+0000 7f52bce00640 1 --2- 172.21.15.28:0/3187669408 >> v2:172.21.15.28:3300/0 conn(0x7f52b80c6350 0x7f52b81559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:27.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.407+0000 7f52bce00640 1 --2- 172.21.15.28:0/3187669408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b814fa80 0x7f52b80c1960 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:27.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.407+0000 7f52bce00640 1 -- 172.21.15.28:0/3187669408 >> 172.21.15.28:0/3187669408 conn(0x7f52b8069e80 msgr2=0x7f52b806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:27.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.408+0000 7f52bce00640 1 -- 172.21.15.28:0/3187669408 shutdown_connections 2024-09-13T16:46:27.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.408+0000 7f52bce00640 1 -- 172.21.15.28:0/3187669408 wait complete. 2024-09-13T16:46:27.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.409+0000 7f52bce00640 1 Processor -- start 2024-09-13T16:46:27.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.409+0000 7f52bce00640 1 -- start start 2024-09-13T16:46:27.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.410+0000 7f52bce00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b8072170 0x7f52b81484c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:27.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.410+0000 7f52bce00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b80c6350 0x7f52b8154e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:27.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.410+0000 7f52bce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b814fa80 0x7f52b81f19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:27.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.410+0000 7f52bce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b8146b30 con 0x7f52b814fa80 2024-09-13T16:46:27.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.410+0000 7f52bce00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b81477f0 con 0x7f52b8072170 2024-09-13T16:46:27.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.410+0000 7f52bce00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b8148a00 con 0x7f52b80c6350 2024-09-13T16:46:27.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.410+0000 7f52b6800640 1 -- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b814fa80 msgr2=0x7f52b81f19f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:46:27.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.410+0000 7f52b6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b814fa80 0x7f52b81f19f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:27.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.411+0000 7f52b5e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b8072170 0x7f52b81484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:27.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.411+0000 7f52b5e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b8072170 0x7f52b81484c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:48138/0 (socket says 172.21.15.28:48138) 2024-09-13T16:46:27.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.411+0000 7f52b5e00640 1 -- 172.21.15.28:0/1083655299 learned_addr learned my addr 172.21.15.28:0/1083655299 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:27.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.411+0000 7f52ad400640 1 --2- 172.21.15.28:0/1083655299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b80c6350 0x7f52b8154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.411+0000 7f52b5e00640 1 -- 172.21.15.28:0/1083655299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b80c6350 msgr2=0x7f52b8154e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.411+0000 7f52b5e00640 1 --2- 172.21.15.28:0/1083655299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b80c6350 0x7f52b8154e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.411+0000 7f52b5e00640 1 -- 172.21.15.28:0/1083655299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b814fa80 msgr2=0x7f52b81f19f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:46:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.411+0000 7f52b5e00640 1 --2- 172.21.15.28:0/1083655299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b814fa80 0x7f52b81f19f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.411+0000 7f52b5e00640 1 -- 172.21.15.28:0/1083655299 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a80094d0 con 0x7f52b8072170 2024-09-13T16:46:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.411+0000 7f52b5e00640 1 --2- 172.21.15.28:0/1083655299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b8072170 0x7f52b81484c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f52a80090b0 tx=0x7f52a80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.412+0000 7f52aea00640 1 -- 172.21.15.28:0/1083655299 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a8032440 con 0x7f52b8072170 2024-09-13T16:46:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.412+0000 7f52aea00640 1 -- 172.21.15.28:0/1083655299 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a8030b80 con 0x7f52b8072170 2024-09-13T16:46:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.412+0000 7f52aea00640 1 -- 172.21.15.28:0/1083655299 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a803b400 con 0x7f52b8072170 2024-09-13T16:46:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.412+0000 7f52bce00640 1 -- 172.21.15.28:0/1083655299 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b80cad80 con 0x7f52b8072170 2024-09-13T16:46:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.412+0000 7f52bce00640 1 -- 172.21.15.28:0/1083655299 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b8146d10 con 0x7f52b8072170 2024-09-13T16:46:27.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.413+0000 7f52bce00640 1 -- 172.21.15.28:0/1083655299 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b80cb340 con 0x7f52b8072170 2024-09-13T16:46:27.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.414+0000 7f52aea00640 1 -- 172.21.15.28:0/1083655299 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f52a8039040 con 0x7f52b8072170 2024-09-13T16:46:27.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.414+0000 7f52aea00640 1 --2- 172.21.15.28:0/1083655299 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f529406abe0 0x7f529406d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:27.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.415+0000 7f52aea00640 1 -- 172.21.15.28:0/1083655299 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f52a80a98e0 con 0x7f52b8072170 2024-09-13T16:46:27.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.415+0000 7f52ad400640 1 --2- 172.21.15.28:0/1083655299 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f529406abe0 0x7f529406d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.415+0000 7f52ad400640 1 --2- 172.21.15.28:0/1083655299 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f529406abe0 0x7f529406d0a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f52a00093d0 tx=0x7f52a0009360 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:27.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.419+0000 7f52aea00640 1 -- 172.21.15.28:0/1083655299 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f52a8074ac0 con 0x7f52b8072170 2024-09-13T16:46:27.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.663+0000 7f52bce00640 1 -- 172.21.15.28:0/1083655299 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f52b806a260 con 0x7f52b8072170 2024-09-13T16:46:27.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.663+0000 7f52aea00640 1 -- 172.21.15.28:0/1083655299 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f52a80780e0 con 0x7f52b8072170 2024-09-13T16:46:27.665 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:46:27.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.665+0000 7f52bce00640 1 -- 172.21.15.28:0/1083655299 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f529406abe0 msgr2=0x7f529406d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:27.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.665+0000 7f52bce00640 1 --2- 172.21.15.28:0/1083655299 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f529406abe0 0x7f529406d0a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f52a00093d0 tx=0x7f52a0009360 comp rx=0 tx=0).stop 2024-09-13T16:46:27.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.665+0000 7f52bce00640 1 -- 172.21.15.28:0/1083655299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b8072170 msgr2=0x7f52b81484c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:27.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.665+0000 7f52bce00640 1 --2- 172.21.15.28:0/1083655299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b8072170 0x7f52b81484c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f52a80090b0 tx=0x7f52a80046c0 comp rx=0 tx=0).stop 2024-09-13T16:46:27.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.665+0000 7f52bce00640 1 -- 172.21.15.28:0/1083655299 shutdown_connections 2024-09-13T16:46:27.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.665+0000 7f52bce00640 1 --2- 172.21.15.28:0/1083655299 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f529406abe0 0x7f529406d0a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:27.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.665+0000 7f52bce00640 1 --2- 172.21.15.28:0/1083655299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b814fa80 0x7f52b81f19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:27.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.665+0000 7f52bce00640 1 --2- 172.21.15.28:0/1083655299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b80c6350 0x7f52b8154e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:27.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.665+0000 7f52bce00640 1 --2- 172.21.15.28:0/1083655299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b8072170 0x7f52b81484c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:27.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.666+0000 7f52bce00640 1 -- 172.21.15.28:0/1083655299 >> 172.21.15.28:0/1083655299 conn(0x7f52b8069e80 msgr2=0x7f52b806d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:27.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.666+0000 7f52bce00640 1 -- 172.21.15.28:0/1083655299 shutdown_connections 2024-09-13T16:46:27.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:27.666+0000 7f52bce00640 1 -- 172.21.15.28:0/1083655299 wait complete. 2024-09-13T16:46:29.124 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:29.125 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 re-readers = 3415845.00 kB/sec 2024-09-13T16:46:29.125 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 re-readers = 3407730.13 kB/sec 2024-09-13T16:46:29.125 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 3415845.00 kB/sec 2024-09-13T16:46:29.125 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 3415845.00 kB/sec 2024-09-13T16:46:29.125 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 3415845.00 kB/sec 2024-09-13T16:46:29.125 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 1048576.00 kB 2024-09-13T16:46:30.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:46:30.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.848+0000 7f4e74a00640 1 Processor -- start 2024-09-13T16:46:30.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.848+0000 7f4e74a00640 1 -- start start 2024-09-13T16:46:30.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.849+0000 7f4e74a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4e701560e0 0x7f4e700c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:30.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.850+0000 7f4e74a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e700c42b0 con 0x7f4e7014fa80 2024-09-13T16:46:30.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.850+0000 7f4e74a00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e700c36e0 con 0x7f4e70072170 2024-09-13T16:46:30.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.850+0000 7f4e74a00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e700c3980 con 0x7f4e701560e0 2024-09-13T16:46:30.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.850+0000 7f4e6de00640 1 -- >> v1:172.21.15.28:6789/0 conn(0x7f4e7014fa80 legacy=0x7f4e700c6ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.28:6789/0 2024-09-13T16:46:30.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.850+0000 7f4e6e800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4e701560e0 0x7f4e700c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:30.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.850+0000 7f4e6e800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4e701560e0 0x7f4e700c1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:35226/0 (socket says 172.21.15.28:35226) 2024-09-13T16:46:30.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.850+0000 7f4e67e00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f4e70072170 0x7f4e701559d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:38942/0 (socket says 172.21.15.28:38942) 2024-09-13T16:46:30.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.850+0000 7f4e6e800640 1 -- 172.21.15.28:0/604421065 learned_addr learned my addr 172.21.15.28:0/604421065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:30.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.850+0000 7f4e6e800640 1 -- 172.21.15.28:0/604421065 >> v1:172.21.15.191:6789/0 conn(0x7f4e70072170 legacy=0x7f4e701559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:30.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.850+0000 7f4e6e800640 1 -- 172.21.15.28:0/604421065 >> v1:172.21.15.28:6789/0 conn(0x7f4e7014fa80 legacy=0x7f4e700c6ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:46:30.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.850+0000 7f4e6e800640 1 -- 172.21.15.28:0/604421065 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e700c3c60 con 0x7f4e701560e0 2024-09-13T16:46:30.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.851+0000 7f4e6e800640 1 --2- 172.21.15.28:0/604421065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e701560e0 0x7f4e700c1960 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4e68009e50 tx=0x7f4e6802a760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c31643d3c9282e3c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:30.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.852+0000 7f4e6d400640 1 -- 172.21.15.28:0/604421065 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e68031600 con 0x7f4e701560e0 2024-09-13T16:46:30.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.852+0000 7f4e6d400640 1 -- 172.21.15.28:0/604421065 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6802d050 con 0x7f4e701560e0 2024-09-13T16:46:30.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.852+0000 7f4e6d400640 1 -- 172.21.15.28:0/604421065 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e68031980 con 0x7f4e701560e0 2024-09-13T16:46:30.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.852+0000 7f4e74a00640 1 -- 172.21.15.28:0/604421065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e701560e0 msgr2=0x7f4e700c1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:30.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.852+0000 7f4e74a00640 1 --2- 172.21.15.28:0/604421065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e701560e0 0x7f4e700c1960 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4e68009e50 tx=0x7f4e6802a760 comp rx=0 tx=0).stop 2024-09-13T16:46:30.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.852+0000 7f4e74a00640 1 -- 172.21.15.28:0/604421065 shutdown_connections 2024-09-13T16:46:30.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.852+0000 7f4e74a00640 1 --2- 172.21.15.28:0/604421065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e701560e0 0x7f4e700c1960 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:30.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.852+0000 7f4e74a00640 1 -- 172.21.15.28:0/604421065 >> 172.21.15.28:0/604421065 conn(0x7f4e70069e80 msgr2=0x7f4e7006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:30.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.852+0000 7f4e74a00640 1 -- 172.21.15.28:0/604421065 shutdown_connections 2024-09-13T16:46:30.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.852+0000 7f4e74a00640 1 -- 172.21.15.28:0/604421065 wait complete. 2024-09-13T16:46:30.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.854+0000 7f4e74a00640 1 Processor -- start 2024-09-13T16:46:30.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.854+0000 7f4e74a00640 1 -- start start 2024-09-13T16:46:30.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.855+0000 7f4e74a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e70072170 0x7f4e701f2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:30.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.855+0000 7f4e74a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e7014fa80 0x7f4e701ef090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:30.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.855+0000 7f4e74a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e701560e0 0x7f4e700c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:30.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.855+0000 7f4e74a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e701eec70 con 0x7f4e70072170 2024-09-13T16:46:30.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.855+0000 7f4e74a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e701ee680 con 0x7f4e701560e0 2024-09-13T16:46:30.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.855+0000 7f4e74a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e701f2b40 con 0x7f4e7014fa80 2024-09-13T16:46:30.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.855+0000 7f4e6de00640 1 -- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e70072170 msgr2=0x7f4e701f2600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:46:30.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.855+0000 7f4e6de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e70072170 0x7f4e701f2600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:30.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.855+0000 7f4e6e800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e701560e0 0x7f4e700c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.855+0000 7f4e67e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e7014fa80 0x7f4e701ef090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.855+0000 7f4e6e800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e701560e0 0x7f4e700c98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:35240/0 (socket says 172.21.15.28:35240) 2024-09-13T16:46:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.855+0000 7f4e67e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e7014fa80 0x7f4e701ef090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:40808/0 (socket says 172.21.15.28:40808) 2024-09-13T16:46:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.855+0000 7f4e6e800640 1 -- 172.21.15.28:0/631304703 learned_addr learned my addr 172.21.15.28:0/631304703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.856+0000 7f4e6e800640 1 -- 172.21.15.28:0/631304703 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e7014fa80 msgr2=0x7f4e701ef090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.856+0000 7f4e6e800640 1 --2- 172.21.15.28:0/631304703 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e7014fa80 0x7f4e701ef090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.856+0000 7f4e6e800640 1 -- 172.21.15.28:0/631304703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e70072170 msgr2=0x7f4e701f2600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:46:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.856+0000 7f4e6e800640 1 --2- 172.21.15.28:0/631304703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e70072170 0x7f4e701f2600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.856+0000 7f4e6e800640 1 -- 172.21.15.28:0/631304703 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e68009b30 con 0x7f4e701560e0 2024-09-13T16:46:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.856+0000 7f4e6e800640 1 --2- 172.21.15.28:0/631304703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e701560e0 0x7f4e700c98a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4e68034040 tx=0x7f4e68004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.856+0000 7f4e66000640 1 -- 172.21.15.28:0/631304703 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6803d860 con 0x7f4e701560e0 2024-09-13T16:46:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.856+0000 7f4e66000640 1 -- 172.21.15.28:0/631304703 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6803d9c0 con 0x7f4e701560e0 2024-09-13T16:46:30.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.856+0000 7f4e74a00640 1 -- 172.21.15.28:0/631304703 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e701dc5c0 con 0x7f4e701560e0 2024-09-13T16:46:30.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.856+0000 7f4e66000640 1 -- 172.21.15.28:0/631304703 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6803dd70 con 0x7f4e701560e0 2024-09-13T16:46:30.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.857+0000 7f4e74a00640 1 -- 172.21.15.28:0/631304703 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e701dc000 con 0x7f4e701560e0 2024-09-13T16:46:30.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.858+0000 7f4e74a00640 1 -- 172.21.15.28:0/631304703 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e701eedb0 con 0x7f4e701560e0 2024-09-13T16:46:30.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.859+0000 7f4e66000640 1 -- 172.21.15.28:0/631304703 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4e6803b400 con 0x7f4e701560e0 2024-09-13T16:46:30.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.859+0000 7f4e66000640 1 --2- 172.21.15.28:0/631304703 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4e4c06abb0 0x7f4e4c06d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:30.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.860+0000 7f4e66000640 1 -- 172.21.15.28:0/631304703 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4e680aa280 con 0x7f4e701560e0 2024-09-13T16:46:30.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.860+0000 7f4e6de00640 1 --2- 172.21.15.28:0/631304703 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4e4c06abb0 0x7f4e4c06d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:30.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.860+0000 7f4e6de00640 1 --2- 172.21.15.28:0/631304703 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4e4c06abb0 0x7f4e4c06d070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4e60006fd0 tx=0x7f4e60006d80 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:30.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:30.864+0000 7f4e66000640 1 -- 172.21.15.28:0/631304703 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4e701eedb0 con 0x7f4e701560e0 2024-09-13T16:46:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:31.113+0000 7f4e74a00640 1 -- 172.21.15.28:0/631304703 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4e701eedb0 con 0x7f4e701560e0 2024-09-13T16:46:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:31.113+0000 7f4e66000640 1 -- 172.21.15.28:0/631304703 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f4e701eedb0 con 0x7f4e701560e0 2024-09-13T16:46:31.115 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:46:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:31.116+0000 7f4e74a00640 1 -- 172.21.15.28:0/631304703 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4e4c06abb0 msgr2=0x7f4e4c06d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:31.116+0000 7f4e74a00640 1 --2- 172.21.15.28:0/631304703 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4e4c06abb0 0x7f4e4c06d070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4e60006fd0 tx=0x7f4e60006d80 comp rx=0 tx=0).stop 2024-09-13T16:46:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:31.116+0000 7f4e74a00640 1 -- 172.21.15.28:0/631304703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e701560e0 msgr2=0x7f4e700c98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:31.116+0000 7f4e74a00640 1 --2- 172.21.15.28:0/631304703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e701560e0 0x7f4e700c98a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4e68034040 tx=0x7f4e68004a40 comp rx=0 tx=0).stop 2024-09-13T16:46:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:31.116+0000 7f4e74a00640 1 -- 172.21.15.28:0/631304703 shutdown_connections 2024-09-13T16:46:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:31.116+0000 7f4e74a00640 1 --2- 172.21.15.28:0/631304703 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4e4c06abb0 0x7f4e4c06d070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:31.116+0000 7f4e74a00640 1 --2- 172.21.15.28:0/631304703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4e701560e0 0x7f4e700c98a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:31.116+0000 7f4e74a00640 1 --2- 172.21.15.28:0/631304703 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4e7014fa80 0x7f4e701ef090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:31.116+0000 7f4e74a00640 1 --2- 172.21.15.28:0/631304703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e70072170 0x7f4e701f2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:31.116+0000 7f4e74a00640 1 -- 172.21.15.28:0/631304703 >> 172.21.15.28:0/631304703 conn(0x7f4e70069e80 msgr2=0x7f4e701e14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:31.116+0000 7f4e74a00640 1 -- 172.21.15.28:0/631304703 shutdown_connections 2024-09-13T16:46:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:31.116+0000 7f4e74a00640 1 -- 172.21.15.28:0/631304703 wait complete. 2024-09-13T16:46:33.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:46:33.879 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:46:34.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:46:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.287+0000 7f8dde000640 1 Processor -- start 2024-09-13T16:46:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.287+0000 7f8dde000640 1 -- start start 2024-09-13T16:46:34.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.288+0000 7f8dde000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8dd8150a70 0x7f8dd80c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:34.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.288+0000 7f8dde000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 0x7f8dd80c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:34.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.288+0000 7f8dde000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd8159f70 con 0x7f8dd80c53a0 2024-09-13T16:46:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.288+0000 7f8dde000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd80c57b0 con 0x7f8dd8150a70 2024-09-13T16:46:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.288+0000 7f8dde000640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd80c64f0 con 0x7f8dd80c6720 2024-09-13T16:46:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.288+0000 7f8dd7400640 1 -- >> v2:172.21.15.28:3300/0 conn(0x7f8dd8150a70 msgr2=0x7f8dd80c1960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:46:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.288+0000 7f8dd7e00640 1 -- >> v1:172.21.15.28:6789/0 conn(0x7f8dd80c53a0 legacy=0x7f8dd8156c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.28:6789/0 2024-09-13T16:46:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.288+0000 7f8dd7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8dd8150a70 0x7f8dd80c1960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.289+0000 7f8dd6a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 0x7f8dd80c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.289+0000 7f8dd6a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 0x7f8dd80c4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:35252/0 (socket says 172.21.15.28:35252) 2024-09-13T16:46:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.289+0000 7f8dd6a00640 1 -- 172.21.15.28:0/1125345718 learned_addr learned my addr 172.21.15.28:0/1125345718 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.289+0000 7f8dd6a00640 1 -- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 msgr2=0x7f8dd80c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T16:46:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.289+0000 7f8dd6a00640 1 -- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 msgr2=0x7f8dd80c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:46:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.289+0000 7f8dd6a00640 1 --2- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 0x7f8dd80c4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:46:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.289+0000 7f8dd6a00640 1 --2- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 0x7f8dd80c4e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.489+0000 7f8dd7400640 1 -- 172.21.15.28:0/1125345718 >> v2:172.21.15.28:3300/0 conn(0x7f8dd8150a70 msgr2=0x7f8dd80c1960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:46:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.489+0000 7f8dd7400640 1 --2- 172.21.15.28:0/1125345718 >> v2:172.21.15.28:3300/0 conn(0x7f8dd8150a70 0x7f8dd80c1960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T16:46:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.489+0000 7f8dd7e00640 1 -- 172.21.15.28:0/1125345718 >> v1:172.21.15.28:6789/0 conn(0x7f8dd80c53a0 legacy=0x7f8dd8156c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.28:6789/0 2024-09-13T16:46:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.490+0000 7f8dd6a00640 1 --2- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 0x7f8dd80c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.490+0000 7f8dd6a00640 1 -- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 msgr2=0x7f8dd80c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:46:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.490+0000 7f8dd6a00640 1 -- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 msgr2=0x7f8dd80c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:46:34.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.490+0000 7f8dd6a00640 1 --2- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 0x7f8dd80c4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:46:34.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.490+0000 7f8dd6a00640 1 --2- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 0x7f8dd80c4e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T16:46:34.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.890+0000 7f8dd7e00640 1 -- 172.21.15.28:0/1125345718 >> v1:172.21.15.28:6789/0 conn(0x7f8dd80c53a0 legacy=0x7f8dd8156c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.28:6789/0 2024-09-13T16:46:34.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.890+0000 7f8dd7400640 1 -- 172.21.15.28:0/1125345718 >> v2:172.21.15.28:3300/0 conn(0x7f8dd8150a70 msgr2=0x7f8dd80c1960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:46:34.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.890+0000 7f8dd7400640 1 --2- 172.21.15.28:0/1125345718 >> v2:172.21.15.28:3300/0 conn(0x7f8dd8150a70 0x7f8dd80c1960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T16:46:34.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.891+0000 7f8dd6a00640 1 --2- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 0x7f8dd80c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:34.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.891+0000 7f8dd6a00640 1 -- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 msgr2=0x7f8dd80c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:46:34.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.891+0000 7f8dd6a00640 1 -- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 msgr2=0x7f8dd80c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:46:34.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.891+0000 7f8dd6a00640 1 --2- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 0x7f8dd80c4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:46:34.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:34.891+0000 7f8dd6a00640 1 --2- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 0x7f8dd80c4e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T16:46:35.646 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:35.646 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:35.646 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:35.646 INFO:tasks.workunit.client.0.smithi028.stdout:iozone test complete. 2024-09-13T16:46:35.649 INFO:tasks.workunit.client.0.smithi028.stderr:+ iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-09-13T16:46:35.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:35.691+0000 7f8dd7e00640 1 -- 172.21.15.28:0/1125345718 >> v1:172.21.15.28:6789/0 conn(0x7f8dd80c53a0 legacy=0x7f8dd8156c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.28:6789/0 2024-09-13T16:46:35.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:35.691+0000 7f8dd7400640 1 -- 172.21.15.28:0/1125345718 >> v2:172.21.15.28:3300/0 conn(0x7f8dd8150a70 msgr2=0x7f8dd80c1960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:46:35.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:35.691+0000 7f8dd7400640 1 --2- 172.21.15.28:0/1125345718 >> v2:172.21.15.28:3300/0 conn(0x7f8dd8150a70 0x7f8dd80c1960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T16:46:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:35.693+0000 7f8dd6a00640 1 --2- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 0x7f8dd80c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:35.693+0000 7f8dd6a00640 1 -- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 msgr2=0x7f8dd80c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:46:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:35.693+0000 7f8dd6a00640 1 -- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 msgr2=0x7f8dd80c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:46:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:35.693+0000 7f8dd6a00640 1 --2- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 0x7f8dd80c4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:46:35.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:35.693+0000 7f8dd6a00640 1 --2- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 0x7f8dd80c4e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T16:46:35.730 INFO:tasks.workunit.client.0.smithi028.stdout: Iozone: Performance Test of File I/O 2024-09-13T16:46:35.730 INFO:tasks.workunit.client.0.smithi028.stdout: Version $Revision: 3.494 $ 2024-09-13T16:46:35.730 INFO:tasks.workunit.client.0.smithi028.stdout: Compiled for 64 bit mode. 2024-09-13T16:46:35.730 INFO:tasks.workunit.client.0.smithi028.stdout: Build: linux-AMD64 2024-09-13T16:46:35.730 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:35.730 INFO:tasks.workunit.client.0.smithi028.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-09-13T16:46:35.730 INFO:tasks.workunit.client.0.smithi028.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-09-13T16:46:35.730 INFO:tasks.workunit.client.0.smithi028.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-09-13T16:46:35.730 INFO:tasks.workunit.client.0.smithi028.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-09-13T16:46:35.730 INFO:tasks.workunit.client.0.smithi028.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-09-13T16:46:35.730 INFO:tasks.workunit.client.0.smithi028.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-09-13T16:46:35.730 INFO:tasks.workunit.client.0.smithi028.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-09-13T16:46:35.731 INFO:tasks.workunit.client.0.smithi028.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-09-13T16:46:35.731 INFO:tasks.workunit.client.0.smithi028.stdout: Alexey Skidanov, Sudhir Kumar. 2024-09-13T16:46:35.731 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:35.731 INFO:tasks.workunit.client.0.smithi028.stdout: Run began: Fri Sep 13 16:46:35 2024 2024-09-13T16:46:35.731 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:46:35.731 INFO:tasks.workunit.client.0.smithi028.stdout: Include close in write timing 2024-09-13T16:46:35.731 INFO:tasks.workunit.client.0.smithi028.stdout: Include fsync in write timing 2024-09-13T16:46:35.731 INFO:tasks.workunit.client.0.smithi028.stdout: File size set to 10485760 kB 2024-09-13T16:46:35.731 INFO:tasks.workunit.client.0.smithi028.stdout: Record Size 1024 kB 2024-09-13T16:46:35.731 INFO:tasks.workunit.client.0.smithi028.stdout: Command line used: iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-09-13T16:46:35.731 INFO:tasks.workunit.client.0.smithi028.stdout: Output is in kBytes/sec 2024-09-13T16:46:35.731 INFO:tasks.workunit.client.0.smithi028.stdout: Time Resolution = 0.000001 seconds. 2024-09-13T16:46:35.732 INFO:tasks.workunit.client.0.smithi028.stdout: Processor cache size set to 1024 kBytes. 2024-09-13T16:46:35.732 INFO:tasks.workunit.client.0.smithi028.stdout: Processor cache line size set to 32 bytes. 2024-09-13T16:46:35.732 INFO:tasks.workunit.client.0.smithi028.stdout: File stride size set to 17 * record size. 2024-09-13T16:46:35.732 INFO:tasks.workunit.client.0.smithi028.stdout: Throughput test with 1 process 2024-09-13T16:46:35.732 INFO:tasks.workunit.client.0.smithi028.stdout: Each process writes a 10485760 kByte file in 1024 kByte records 2024-09-13T16:46:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.289+0000 7f8dd4c00640 1 -- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 msgr2=0x7f8dd80c4e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:46:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.289+0000 7f8dd4c00640 1 --2- 172.21.15.28:0/1125345718 >> v2:172.21.15.191:3300/0 conn(0x7f8dd80c6720 0x7f8dd80c4e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.289+0000 7f8dd4c00640 1 -- 172.21.15.28:0/1125345718 >> v2:172.21.15.28:3300/0 conn(0x7f8dd8150a70 msgr2=0x7f8dd80c1960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:46:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.289+0000 7f8dd4c00640 1 --2- 172.21.15.28:0/1125345718 >> v2:172.21.15.28:3300/0 conn(0x7f8dd8150a70 0x7f8dd80c1960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.289+0000 7f8dd4c00640 1 -- 172.21.15.28:0/1125345718 >> v1:172.21.15.28:6789/0 conn(0x7f8dd80c53a0 legacy=0x7f8dd8156c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:46:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.289+0000 7f8dd4c00640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8db80036a0 0x7f8db8005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.289+0000 7f8dd4c00640 1 -- 172.21.15.28:0/1125345718 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd80c1ea0 con 0x7f8db80061c0 2024-09-13T16:46:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.289+0000 7f8dd4c00640 1 -- 172.21.15.28:0/1125345718 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd80c57b0 con 0x7f8db80036a0 2024-09-13T16:46:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.289+0000 7f8dd7400640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8db80036a0 0x7f8db8005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.289+0000 7f8dd7400640 1 -- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8db80036a0 msgr2=0x7f8db8005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:46:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.289+0000 7f8dd7400640 1 -- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8db80036a0 msgr2=0x7f8db8005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:46:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.289+0000 7f8dd7400640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8db80036a0 0x7f8db8005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:46:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.289+0000 7f8dd7400640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8db80036a0 0x7f8db8005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:37.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.490+0000 7f8dd7400640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8db80036a0 0x7f8db8005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:37.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.490+0000 7f8dd7400640 1 -- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8db80036a0 msgr2=0x7f8db8005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:46:37.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.490+0000 7f8dd7400640 1 -- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8db80036a0 msgr2=0x7f8db8005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:46:37.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.490+0000 7f8dd7400640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8db80036a0 0x7f8db8005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:46:37.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.490+0000 7f8dd7400640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8db80036a0 0x7f8db8005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T16:46:37.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.722+0000 7f8dd6000640 1 -- 172.21.15.28:0/1125345718 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621868535 0 0) 0x7f8dd80c1ea0 con 0x7f8db80061c0 2024-09-13T16:46:37.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.722+0000 7f8dd6000640 1 -- 172.21.15.28:0/1125345718 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dbc003520 con 0x7f8db80061c0 2024-09-13T16:46:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.725+0000 7f8dd6000640 1 -- 172.21.15.28:0/1125345718 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2656512521 0 0) 0x7f8dbc003520 con 0x7f8db80061c0 2024-09-13T16:46:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.725+0000 7f8dd6000640 1 -- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8db80036a0 msgr2=0x7f8db8005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:46:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.725+0000 7f8dd6000640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8db80036a0 0x7f8db8005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.725+0000 7f8dd6000640 1 -- 172.21.15.28:0/1125345718 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd8159f70 con 0x7f8db80061c0 2024-09-13T16:46:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.725+0000 7f8dd6000640 1 -- 172.21.15.28:0/1125345718 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1654382301 0 0) 0x7f8dc400c570 con 0x7f8db80061c0 2024-09-13T16:46:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.725+0000 7f8dd6000640 1 -- 172.21.15.28:0/1125345718 >> v1:172.21.15.191:6789/0 conn(0x7f8db80061c0 legacy=0x7f8db8008680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.725+0000 7f8dd7e00640 1 -- 172.21.15.28:0/1125345718 reap_dead start 2024-09-13T16:46:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.725+0000 7f8dd6000640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dbc003970 0x7f8dbc003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.725+0000 7f8dd6000640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dbc0047c0 0x7f8dbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.725+0000 7f8dd6000640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dbc0051d0 0x7f8dbc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.725+0000 7f8dd6000640 1 -- 172.21.15.28:0/1125345718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd80c57b0 con 0x7f8dbc0051d0 2024-09-13T16:46:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.725+0000 7f8dd6000640 1 -- 172.21.15.28:0/1125345718 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc00c100 con 0x7f8dbc003970 2024-09-13T16:46:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.725+0000 7f8dd6000640 1 -- 172.21.15.28:0/1125345718 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc00c2c0 con 0x7f8dbc0047c0 2024-09-13T16:46:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.725+0000 7f8dd7e00640 1 -- 172.21.15.28:0/1125345718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dbc0051d0 msgr2=0x7f8dbc00b980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:46:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.725+0000 7f8dd7400640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dbc003970 0x7f8dbc003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.725+0000 7f8dd7e00640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dbc0051d0 0x7f8dbc00b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.726+0000 7f8dd6a00640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dbc0047c0 0x7f8dbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.726+0000 7f8dd7400640 1 -- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dbc0047c0 msgr2=0x7f8dbc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:46:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.726+0000 7f8dd7400640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dbc0047c0 0x7f8dbc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.726+0000 7f8dd7400640 1 -- 172.21.15.28:0/1125345718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dbc0051d0 msgr2=0x7f8dbc00b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:46:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.726+0000 7f8dd7400640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dbc0051d0 0x7f8dbc00b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.726+0000 7f8dd7400640 1 -- 172.21.15.28:0/1125345718 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dbc003520 con 0x7f8dbc003970 2024-09-13T16:46:37.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.727+0000 7f8dd7400640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dbc003970 0x7f8dbc003d70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8003980 tx=0x7f8dbc0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10e84a8bc14d4f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:37.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.728+0000 7f8dd6000640 1 -- 172.21.15.28:0/1125345718 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc802c700 con 0x7f8dbc003970 2024-09-13T16:46:37.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.728+0000 7f8dd6000640 1 -- 172.21.15.28:0/1125345718 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc802c860 con 0x7f8dbc003970 2024-09-13T16:46:37.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.728+0000 7f8dd6000640 1 -- 172.21.15.28:0/1125345718 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc802cc10 con 0x7f8dbc003970 2024-09-13T16:46:37.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.729+0000 7f8dde000640 1 -- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dbc003970 msgr2=0x7f8dbc003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:37.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.729+0000 7f8dde000640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dbc003970 0x7f8dbc003d70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8003980 tx=0x7f8dbc0057f0 comp rx=0 tx=0).stop 2024-09-13T16:46:37.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.729+0000 7f8dde000640 1 -- 172.21.15.28:0/1125345718 shutdown_connections 2024-09-13T16:46:37.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.729+0000 7f8dde000640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dbc0051d0 0x7f8dbc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:37.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.729+0000 7f8dde000640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dbc0047c0 0x7f8dbc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:37.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.729+0000 7f8dde000640 1 --2- 172.21.15.28:0/1125345718 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dbc003970 0x7f8dbc003d70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:37.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.729+0000 7f8dde000640 1 -- 172.21.15.28:0/1125345718 >> 172.21.15.28:0/1125345718 conn(0x7f8dd8069e80 msgr2=0x7f8dd806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:37.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.729+0000 7f8dde000640 1 -- 172.21.15.28:0/1125345718 shutdown_connections 2024-09-13T16:46:37.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.729+0000 7f8dde000640 1 -- 172.21.15.28:0/1125345718 wait complete. 2024-09-13T16:46:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.731+0000 7f8dde000640 1 Processor -- start 2024-09-13T16:46:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.732+0000 7f8dde000640 1 -- start start 2024-09-13T16:46:37.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.732+0000 7f8dde000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dbc003970 0x7f8dd80711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:37.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dde000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dbc0047c0 0x7f8dd81f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:37.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dde000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dbc0051d0 0x7f8dd81f2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:37.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dde000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd81f1a30 con 0x7f8dbc0051d0 2024-09-13T16:46:37.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dde000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd8159970 con 0x7f8dbc0047c0 2024-09-13T16:46:37.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dde000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd8157f10 con 0x7f8dbc003970 2024-09-13T16:46:37.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dd7e00640 1 -- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dbc0051d0 msgr2=0x7f8dd81f2400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:46:37.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dd7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dbc0051d0 0x7f8dd81f2400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dd6a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dbc0047c0 0x7f8dd81f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dd7400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dbc003970 0x7f8dd80711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dd7400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dbc003970 0x7f8dd80711e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:40840/0 (socket says 172.21.15.28:40840) 2024-09-13T16:46:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dd6a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dbc0047c0 0x7f8dd81f1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:35302/0 (socket says 172.21.15.28:35302) 2024-09-13T16:46:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dd7400640 1 -- 172.21.15.28:0/636855756 learned_addr learned my addr 172.21.15.28:0/636855756 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dd7400640 1 -- 172.21.15.28:0/636855756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dbc0047c0 msgr2=0x7f8dd81f1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dd7400640 1 --2- 172.21.15.28:0/636855756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dbc0047c0 0x7f8dd81f1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dd7400640 1 -- 172.21.15.28:0/636855756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dbc0051d0 msgr2=0x7f8dd81f2400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:46:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dd7400640 1 --2- 172.21.15.28:0/636855756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dbc0051d0 0x7f8dd81f2400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dd7400640 1 -- 172.21.15.28:0/636855756 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd8159bc0 con 0x7f8dbc003970 2024-09-13T16:46:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.733+0000 7f8dd6a00640 1 --2- 172.21.15.28:0/636855756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dbc0047c0 0x7f8dd81f1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:46:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.734+0000 7f8dd7400640 1 --2- 172.21.15.28:0/636855756 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dbc003970 0x7f8dd80711e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8005700 tx=0x7f8dc8005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:37.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.734+0000 7f8dcf400640 1 -- 172.21.15.28:0/636855756 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8037c90 con 0x7f8dbc003970 2024-09-13T16:46:37.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.734+0000 7f8dcf400640 1 -- 172.21.15.28:0/636855756 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc8037df0 con 0x7f8dbc003970 2024-09-13T16:46:37.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.734+0000 7f8dcf400640 1 -- 172.21.15.28:0/636855756 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8041660 con 0x7f8dbc003970 2024-09-13T16:46:37.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.734+0000 7f8dde000640 1 -- 172.21.15.28:0/636855756 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd8153ce0 con 0x7f8dbc003970 2024-09-13T16:46:37.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.734+0000 7f8dde000640 1 -- 172.21.15.28:0/636855756 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd8154c90 con 0x7f8dbc003970 2024-09-13T16:46:37.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.736+0000 7f8dde000640 1 -- 172.21.15.28:0/636855756 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd815ad60 con 0x7f8dbc003970 2024-09-13T16:46:37.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.736+0000 7f8dcf400640 1 -- 172.21.15.28:0/636855756 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8dc80417c0 con 0x7f8dbc003970 2024-09-13T16:46:37.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.737+0000 7f8dcf400640 1 --2- 172.21.15.28:0/636855756 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8db806b390 0x7f8db806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:37.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.737+0000 7f8dcf400640 1 -- 172.21.15.28:0/636855756 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8dc80a9f00 con 0x7f8dbc003970 2024-09-13T16:46:37.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.737+0000 7f8dd6a00640 1 --2- 172.21.15.28:0/636855756 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8db806b390 0x7f8db806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:37.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.738+0000 7f8dd6a00640 1 --2- 172.21.15.28:0/636855756 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8db806b390 0x7f8db806d850 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8dd81566c0 tx=0x7f8dc4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:37.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.742+0000 7f8dcf400640 1 -- 172.21.15.28:0/636855756 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8dc8075110 con 0x7f8dbc003970 2024-09-13T16:46:37.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.990+0000 7f8dde000640 1 -- 172.21.15.28:0/636855756 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8dd81574a0 con 0x7f8dbc003970 2024-09-13T16:46:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.991+0000 7f8dcf400640 1 -- 172.21.15.28:0/636855756 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f8dc8078730 con 0x7f8dbc003970 2024-09-13T16:46:37.992 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":18,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:46:37.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.993+0000 7f8dde000640 1 -- 172.21.15.28:0/636855756 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8db806b390 msgr2=0x7f8db806d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:37.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.993+0000 7f8dde000640 1 --2- 172.21.15.28:0/636855756 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8db806b390 0x7f8db806d850 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8dd81566c0 tx=0x7f8dc4008040 comp rx=0 tx=0).stop 2024-09-13T16:46:37.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.993+0000 7f8dde000640 1 -- 172.21.15.28:0/636855756 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dbc003970 msgr2=0x7f8dd80711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:37.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.993+0000 7f8dde000640 1 --2- 172.21.15.28:0/636855756 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dbc003970 0x7f8dd80711e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8005700 tx=0x7f8dc8005bd0 comp rx=0 tx=0).stop 2024-09-13T16:46:37.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.993+0000 7f8dde000640 1 -- 172.21.15.28:0/636855756 shutdown_connections 2024-09-13T16:46:37.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.993+0000 7f8dde000640 1 --2- 172.21.15.28:0/636855756 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8db806b390 0x7f8db806d850 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:37.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.993+0000 7f8dde000640 1 --2- 172.21.15.28:0/636855756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dbc0051d0 0x7f8dd81f2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:37.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.993+0000 7f8dde000640 1 --2- 172.21.15.28:0/636855756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dbc0047c0 0x7f8dd81f1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:37.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.993+0000 7f8dde000640 1 --2- 172.21.15.28:0/636855756 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dbc003970 0x7f8dd80711e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:37.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.993+0000 7f8dde000640 1 -- 172.21.15.28:0/636855756 >> 172.21.15.28:0/636855756 conn(0x7f8dd8069e80 msgr2=0x7f8dd806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:37.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.993+0000 7f8dde000640 1 -- 172.21.15.28:0/636855756 shutdown_connections 2024-09-13T16:46:37.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:37.993+0000 7f8dde000640 1 -- 172.21.15.28:0/636855756 wait complete. 2024-09-13T16:46:38.011 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-13T16:46:38.012 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-13T16:46:38.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T16:46:38.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.167+0000 7f62d7c00640 1 Processor -- start 2024-09-13T16:46:38.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.167+0000 7f62d7c00640 1 -- start start 2024-09-13T16:46:38.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.168+0000 7f62d7c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62d00c67b0 0x7f62d006d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:38.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.168+0000 7f62d7c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d015cbc0 con 0x7f62d006d770 2024-09-13T16:46:38.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.168+0000 7f62d7c00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d00c5a50 con 0x7f62d0152d50 2024-09-13T16:46:38.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.168+0000 7f62d7c00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d015c080 con 0x7f62d00c67b0 2024-09-13T16:46:38.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.168+0000 7f62d5c00640 1 -- >> v1:172.21.15.28:6789/0 conn(0x7f62d006d770 legacy=0x7f62d006fe80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.28:6789/0 2024-09-13T16:46:38.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.169+0000 7f62cfe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62d00c67b0 0x7f62d006d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:38.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.169+0000 7f62d5200640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f62d0152d50 0x7f62d00c5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:38968/0 (socket says 172.21.15.28:38968) 2024-09-13T16:46:38.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.169+0000 7f62d5200640 1 -- 172.21.15.28:0/90517474 learned_addr learned my addr 172.21.15.28:0/90517474 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:46:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.169+0000 7f62cf400640 1 -- 172.21.15.28:0/90517474 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933565760 0 0) 0x7f62d00c5a50 con 0x7f62d0152d50 2024-09-13T16:46:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.169+0000 7f62cf400640 1 -- 172.21.15.28:0/90517474 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62b8003520 con 0x7f62d0152d50 2024-09-13T16:46:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.169+0000 7f62cfe00640 1 -- 172.21.15.28:0/90517474 >> v1:172.21.15.191:6789/0 conn(0x7f62d0152d50 legacy=0x7f62d00c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.169+0000 7f62cfe00640 1 -- 172.21.15.28:0/90517474 >> v1:172.21.15.28:6789/0 conn(0x7f62d006d770 legacy=0x7f62d006fe80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:46:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.169+0000 7f62cfe00640 1 -- 172.21.15.28:0/90517474 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d00c5d10 con 0x7f62d00c67b0 2024-09-13T16:46:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.170+0000 7f62cfe00640 1 --2- 172.21.15.28:0/90517474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62d00c67b0 0x7f62d006d230 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f62bc0097d0 tx=0x7f62bc02a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77b327e047332b6c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:38.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.170+0000 7f62cf400640 1 -- 172.21.15.28:0/90517474 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc032440 con 0x7f62d00c67b0 2024-09-13T16:46:38.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.170+0000 7f62cf400640 1 -- 172.21.15.28:0/90517474 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62bc0325a0 con 0x7f62d00c67b0 2024-09-13T16:46:38.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.170+0000 7f62cf400640 1 -- 172.21.15.28:0/90517474 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc032910 con 0x7f62d00c67b0 2024-09-13T16:46:38.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.171+0000 7f62d7c00640 1 -- 172.21.15.28:0/90517474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62d00c67b0 msgr2=0x7f62d006d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:38.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.171+0000 7f62d7c00640 1 --2- 172.21.15.28:0/90517474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62d00c67b0 0x7f62d006d230 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f62bc0097d0 tx=0x7f62bc02a250 comp rx=0 tx=0).stop 2024-09-13T16:46:38.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.171+0000 7f62d7c00640 1 -- 172.21.15.28:0/90517474 shutdown_connections 2024-09-13T16:46:38.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.171+0000 7f62d7c00640 1 --2- 172.21.15.28:0/90517474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62d00c67b0 0x7f62d006d230 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:38.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.171+0000 7f62d7c00640 1 -- 172.21.15.28:0/90517474 >> 172.21.15.28:0/90517474 conn(0x7f62d00ca6b0 msgr2=0x7f62d00caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:38.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.171+0000 7f62d7c00640 1 -- 172.21.15.28:0/90517474 shutdown_connections 2024-09-13T16:46:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.171+0000 7f62d7c00640 1 -- 172.21.15.28:0/90517474 wait complete. 2024-09-13T16:46:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.173+0000 7f62d7c00640 1 Processor -- start 2024-09-13T16:46:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.173+0000 7f62d7c00640 1 -- start start 2024-09-13T16:46:38.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62d7c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62d006d770 0x7f62d01e2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:38.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62d7c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62d00c67b0 0x7f62d01e22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:38.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62d7c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d0152d50 0x7f62d0155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:38.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62d7c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d01e2810 con 0x7f62d006d770 2024-09-13T16:46:38.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62d7c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d01e38d0 con 0x7f62d0152d50 2024-09-13T16:46:38.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62d7c00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d01f20e0 con 0x7f62d00c67b0 2024-09-13T16:46:38.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62d5200640 1 -- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62d006d770 msgr2=0x7f62d01e2aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:46:38.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62d5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62d006d770 0x7f62d01e2aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:38.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62cfe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62d00c67b0 0x7f62d01e22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:38.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62d5c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d0152d50 0x7f62d0155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:38.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62cfe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62d00c67b0 0x7f62d01e22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:40850/0 (socket says 172.21.15.28:40850) 2024-09-13T16:46:38.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62cfe00640 1 -- 172.21.15.28:0/1181110267 learned_addr learned my addr 172.21.15.28:0/1181110267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:38.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62d5c00640 1 -- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62d00c67b0 msgr2=0x7f62d01e22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:38.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62d5c00640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62d00c67b0 0x7f62d01e22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:38.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62d5c00640 1 -- 172.21.15.28:0/1181110267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62d006d770 msgr2=0x7f62d01e2aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:46:38.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62d5c00640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62d006d770 0x7f62d01e2aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:38.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62d5c00640 1 -- 172.21.15.28:0/1181110267 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc009400 con 0x7f62d0152d50 2024-09-13T16:46:38.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.174+0000 7f62cfe00640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62d00c67b0 0x7f62d01e22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:46:38.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.175+0000 7f62d5c00640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d0152d50 0x7f62d0155e10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f62c800bd90 tx=0x7f62c8009eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:38.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.175+0000 7f62ccc00640 1 -- 172.21.15.28:0/1181110267 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c80064e0 con 0x7f62d0152d50 2024-09-13T16:46:38.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.175+0000 7f62ccc00640 1 -- 172.21.15.28:0/1181110267 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c800f070 con 0x7f62d0152d50 2024-09-13T16:46:38.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.175+0000 7f62d7c00640 1 -- 172.21.15.28:0/1181110267 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d01f23a0 con 0x7f62d0152d50 2024-09-13T16:46:38.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.175+0000 7f62ccc00640 1 -- 172.21.15.28:0/1181110267 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c80068c0 con 0x7f62d0152d50 2024-09-13T16:46:38.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.175+0000 7f62d7c00640 1 -- 172.21.15.28:0/1181110267 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d01f2960 con 0x7f62d0152d50 2024-09-13T16:46:38.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.176+0000 7f62d7c00640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d01f2af0 0x7f62d01e3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:38.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.176+0000 7f62d7c00640 1 -- 172.21.15.28:0/1181110267 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d0066c20 con 0x7f62d01f2af0 2024-09-13T16:46:38.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.176+0000 7f62d5200640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d01f2af0 0x7f62d01e3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:38.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.177+0000 7f62d5200640 1 -- 172.21.15.28:0/1181110267 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62d0072510 con 0x7f62d01f2af0 2024-09-13T16:46:38.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.177+0000 7f62d5200640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d01f2af0 0x7f62d01e3d60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f62c4004870 tx=0x7f62c400c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:38.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.177+0000 7f62ccc00640 1 -- 172.21.15.28:0/1181110267 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c4005ce0 con 0x7f62d01f2af0 2024-09-13T16:46:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.178+0000 7f62ccc00640 1 -- 172.21.15.28:0/1181110267 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f62c8006a20 con 0x7f62d0152d50 2024-09-13T16:46:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.178+0000 7f62ccc00640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f62b006aca0 0x7f62b006d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.178+0000 7f62ccc00640 1 -- 172.21.15.28:0/1181110267 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f62c808a0e0 con 0x7f62d0152d50 2024-09-13T16:46:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.179+0000 7f62ccc00640 1 -- 172.21.15.28:0/1181110267 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f62c4005e40 con 0x7f62d01f2af0 2024-09-13T16:46:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.179+0000 7f62ccc00640 1 -- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d01f2af0 msgr2=0x7f62d01e3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.179+0000 7f62ccc00640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d01f2af0 0x7f62d01e3d60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f62c4004870 tx=0x7f62c400c8e0 comp rx=0 tx=0).stop 2024-09-13T16:46:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.179+0000 7f62ccc00640 1 -- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d01f2af0 msgr2=0x7f62d01e3d60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:46:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.179+0000 7f62ccc00640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d01f2af0 0x7f62d01e3d60 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f62c4004870 tx=0x7f62c400c8e0 comp rx=0 tx=0).stop 2024-09-13T16:46:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.179+0000 7f62cfe00640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f62b006aca0 0x7f62b006d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.179+0000 7f62cfe00640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f62b006aca0 0x7f62b006d160 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f62d01e5ed0 tx=0x7f62bc02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.185+0000 7f62d7c00640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d0155230 0x7f62d01e42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.185+0000 7f62d7c00640 1 -- 172.21.15.28:0/1181110267 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d015c850 con 0x7f62d0155230 2024-09-13T16:46:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.186+0000 7f62d5200640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d0155230 0x7f62d01e42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.186+0000 7f62d5200640 1 -- 172.21.15.28:0/1181110267 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f62d015c570 con 0x7f62d0155230 2024-09-13T16:46:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.186+0000 7f62d5200640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d0155230 0x7f62d01e42a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f62c4004810 tx=0x7f62c4001150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.186+0000 7f62ccc00640 1 -- 172.21.15.28:0/1181110267 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c4021070 con 0x7f62d0155230 2024-09-13T16:46:38.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.187+0000 7f62ccc00640 1 -- 172.21.15.28:0/1181110267 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4780 (secure 0 0 0) 0x7f62c4004870 con 0x7f62d0155230 2024-09-13T16:46:38.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.187+0000 7f62ccc00640 1 -- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d0155230 msgr2=0x7f62d01e42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:38.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.187+0000 7f62ccc00640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d0155230 0x7f62d01e42a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f62c4004810 tx=0x7f62c4001150 comp rx=0 tx=0).stop 2024-09-13T16:46:38.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.187+0000 7f62ccc00640 1 -- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d0155230 msgr2=0x7f62d01e42a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:46:38.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.187+0000 7f62ccc00640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d0155230 0x7f62d01e42a0 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f62c4004810 tx=0x7f62c4001150 comp rx=0 tx=0).stop 2024-09-13T16:46:38.189 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:46:38.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:46:38.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:46:38.189 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "leader", 2024-09-13T16:46:38.189 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 18, 2024-09-13T16:46:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:46:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:46:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:46:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 0, 2024-09-13T16:46:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:46:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:46:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:46:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:38.190 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:46:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:46:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:38.191 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:46:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:46:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:46:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:46:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:46:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:46:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:46:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:46:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:46:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:46:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:46:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:46:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:46:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:46:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:46:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:46:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:46:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:38.193 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:46:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:46:38.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:46:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:46:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:46:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:46:38.195 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:46:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:46:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:46:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:46:38.196 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:46:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:46:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:38.197 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:46:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:46:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:46:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:46:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:38.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:46:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:46:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:46:38.199 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:46:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:46:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:46:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:38.200 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:46:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:46:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:46:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:46:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:38.201 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:46:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:46:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:46:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.202 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:46:38.202 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:46:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.187+0000 7f62d7c00640 1 -- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f62b006aca0 msgr2=0x7f62b006d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.187+0000 7f62d7c00640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f62b006aca0 0x7f62b006d160 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f62d01e5ed0 tx=0x7f62bc02e000 comp rx=0 tx=0).stop 2024-09-13T16:46:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.187+0000 7f62d7c00640 1 -- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d0152d50 msgr2=0x7f62d0155e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.187+0000 7f62d7c00640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d0152d50 0x7f62d0155e10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f62c800bd90 tx=0x7f62c8009eb0 comp rx=0 tx=0).stop 2024-09-13T16:46:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.187+0000 7f62d5c00640 1 -- 172.21.15.28:0/1181110267 reap_dead start 2024-09-13T16:46:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.187+0000 7f62d7c00640 1 -- 172.21.15.28:0/1181110267 shutdown_connections 2024-09-13T16:46:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.187+0000 7f62d7c00640 1 --2- 172.21.15.28:0/1181110267 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62d0152d50 0x7f62d0155e10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.187+0000 7f62d7c00640 1 -- 172.21.15.28:0/1181110267 >> 172.21.15.28:0/1181110267 conn(0x7f62d00ca6b0 msgr2=0x7f62d006fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.187+0000 7f62d7c00640 1 -- 172.21.15.28:0/1181110267 shutdown_connections 2024-09-13T16:46:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.187+0000 7f62d7c00640 1 -- 172.21.15.28:0/1181110267 wait complete. 2024-09-13T16:46:38.204 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-13T16:46:38.205 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-13T16:46:38.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T16:46:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.360+0000 7f8b4bc00640 1 Processor -- start 2024-09-13T16:46:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.360+0000 7f8b4bc00640 1 -- start start 2024-09-13T16:46:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.361+0000 7f8b4bc00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8b4415a440 0x7f8b440c5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.361+0000 7f8b4bc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b440c1d80 con 0x7f8b4415a060 2024-09-13T16:46:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.361+0000 7f8b4bc00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b4415a920 con 0x7f8b4406dcb0 2024-09-13T16:46:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.361+0000 7f8b4bc00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b4415ac00 con 0x7f8b4415a440 2024-09-13T16:46:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.361+0000 7f8b49200640 1 -- >> v1:172.21.15.28:6789/0 conn(0x7f8b4415a060 legacy=0x7f8b4406d5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.28:6789/0 2024-09-13T16:46:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.362+0000 7f8b43e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8b4415a440 0x7f8b440c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.362+0000 7f8b49c00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f8b4406dcb0 0x7f8b441504e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:38976/0 (socket says 172.21.15.28:38976) 2024-09-13T16:46:38.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.362+0000 7f8b49c00640 1 -- 172.21.15.28:0/3396296860 learned_addr learned my addr 172.21.15.28:0/3396296860 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:46:38.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.362+0000 7f8b43e00640 1 -- 172.21.15.28:0/3396296860 >> v1:172.21.15.191:6789/0 conn(0x7f8b4406dcb0 legacy=0x7f8b441504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:38.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.362+0000 7f8b43e00640 1 -- 172.21.15.28:0/3396296860 >> v1:172.21.15.28:6789/0 conn(0x7f8b4415a060 legacy=0x7f8b4406d5a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:46:38.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.362+0000 7f8b43e00640 1 -- 172.21.15.28:0/3396296860 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b440c2d50 con 0x7f8b4415a440 2024-09-13T16:46:38.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.363+0000 7f8b43e00640 1 --2- 172.21.15.28:0/3396296860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b4415a440 0x7f8b440c5500 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8b300097a0 tx=0x7f8b3002a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2c22891a6e586d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:38.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.363+0000 7f8b43400640 1 -- 172.21.15.28:0/3396296860 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30032440 con 0x7f8b4415a440 2024-09-13T16:46:38.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.363+0000 7f8b43400640 1 -- 172.21.15.28:0/3396296860 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b300325a0 con 0x7f8b4415a440 2024-09-13T16:46:38.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.363+0000 7f8b43400640 1 -- 172.21.15.28:0/3396296860 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30032930 con 0x7f8b4415a440 2024-09-13T16:46:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.364+0000 7f8b4bc00640 1 -- 172.21.15.28:0/3396296860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b4415a440 msgr2=0x7f8b440c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.364+0000 7f8b4bc00640 1 --2- 172.21.15.28:0/3396296860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b4415a440 0x7f8b440c5500 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8b300097a0 tx=0x7f8b3002a1e0 comp rx=0 tx=0).stop 2024-09-13T16:46:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.364+0000 7f8b4bc00640 1 -- 172.21.15.28:0/3396296860 shutdown_connections 2024-09-13T16:46:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.364+0000 7f8b4bc00640 1 --2- 172.21.15.28:0/3396296860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b4415a440 0x7f8b440c5500 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.364+0000 7f8b4bc00640 1 -- 172.21.15.28:0/3396296860 >> 172.21.15.28:0/3396296860 conn(0x7f8b440ca6b0 msgr2=0x7f8b440caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.364+0000 7f8b4bc00640 1 -- 172.21.15.28:0/3396296860 shutdown_connections 2024-09-13T16:46:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.364+0000 7f8b4bc00640 1 -- 172.21.15.28:0/3396296860 wait complete. 2024-09-13T16:46:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.365+0000 7f8b4bc00640 1 Processor -- start 2024-09-13T16:46:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.366+0000 7f8b4bc00640 1 -- start start 2024-09-13T16:46:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.366+0000 7f8b4bc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b4406dcb0 0x7f8b441e2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.366+0000 7f8b4bc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b4415a060 0x7f8b44158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.366+0000 7f8b4bc00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b4415a440 0x7f8b44158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.366+0000 7f8b4bc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b441ee880 con 0x7f8b4415a060 2024-09-13T16:46:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.366+0000 7f8b4bc00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b441eea70 con 0x7f8b4406dcb0 2024-09-13T16:46:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.366+0000 7f8b4bc00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b441eed50 con 0x7f8b4415a440 2024-09-13T16:46:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.366+0000 7f8b43e00640 1 -- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b4415a060 msgr2=0x7f8b44158740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:46:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.367+0000 7f8b43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b4415a060 0x7f8b44158740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.367+0000 7f8b49200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b4406dcb0 0x7f8b441e2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.367+0000 7f8b49c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b4415a440 0x7f8b44158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.367+0000 7f8b49200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b4406dcb0 0x7f8b441e2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:35330/0 (socket says 172.21.15.28:35330) 2024-09-13T16:46:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.367+0000 7f8b49200640 1 -- 172.21.15.28:0/3466524878 learned_addr learned my addr 172.21.15.28:0/3466524878 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.367+0000 7f8b49200640 1 -- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b4415a440 msgr2=0x7f8b44158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.367+0000 7f8b49200640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b4415a440 0x7f8b44158c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.367+0000 7f8b49200640 1 -- 172.21.15.28:0/3466524878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b4415a060 msgr2=0x7f8b44158740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:46:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.367+0000 7f8b49200640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b4415a060 0x7f8b44158740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.367+0000 7f8b49200640 1 -- 172.21.15.28:0/3466524878 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b30009400 con 0x7f8b4406dcb0 2024-09-13T16:46:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.368+0000 7f8b49200640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b4406dcb0 0x7f8b441e2600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8b3800b580 tx=0x7f8b3800ba40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.368+0000 7f8b40c00640 1 -- 172.21.15.28:0/3466524878 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b38003dc0 con 0x7f8b4406dcb0 2024-09-13T16:46:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.368+0000 7f8b40c00640 1 -- 172.21.15.28:0/3466524878 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b38005070 con 0x7f8b4406dcb0 2024-09-13T16:46:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.368+0000 7f8b4bc00640 1 -- 172.21.15.28:0/3466524878 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b441ef030 con 0x7f8b4406dcb0 2024-09-13T16:46:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.368+0000 7f8b40c00640 1 -- 172.21.15.28:0/3466524878 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b38005420 con 0x7f8b4406dcb0 2024-09-13T16:46:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.368+0000 7f8b4bc00640 1 -- 172.21.15.28:0/3466524878 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b441ef5f0 con 0x7f8b4406dcb0 2024-09-13T16:46:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.369+0000 7f8b4bc00640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b4414d8c0 0x7f8b441e0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.369+0000 7f8b4bc00640 1 -- 172.21.15.28:0/3466524878 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b44066c20 con 0x7f8b4414d8c0 2024-09-13T16:46:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.369+0000 7f8b43e00640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b4414d8c0 0x7f8b441e0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.369+0000 7f8b43e00640 1 -- 172.21.15.28:0/3466524878 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b440c5220 con 0x7f8b4414d8c0 2024-09-13T16:46:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.370+0000 7f8b43e00640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b4414d8c0 0x7f8b441e0d40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8b4406ca00 tx=0x7f8b300046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.370+0000 7f8b40c00640 1 -- 172.21.15.28:0/3466524878 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8b38005580 con 0x7f8b4406dcb0 2024-09-13T16:46:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.371+0000 7f8b40c00640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8b2406abb0 0x7f8b2406d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.371+0000 7f8b40c00640 1 -- 172.21.15.28:0/3466524878 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8b38089480 con 0x7f8b4406dcb0 2024-09-13T16:46:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.371+0000 7f8b49c00640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8b2406abb0 0x7f8b2406d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.371+0000 7f8b40c00640 1 -- 172.21.15.28:0/3466524878 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30004220 con 0x7f8b4414d8c0 2024-09-13T16:46:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.371+0000 7f8b40c00640 1 -- 172.21.15.28:0/3466524878 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f8b30004380 con 0x7f8b4414d8c0 2024-09-13T16:46:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.371+0000 7f8b40c00640 1 -- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b4414d8c0 msgr2=0x7f8b441e0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.371+0000 7f8b40c00640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b4414d8c0 0x7f8b441e0d40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8b4406ca00 tx=0x7f8b300046c0 comp rx=0 tx=0).stop 2024-09-13T16:46:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.371+0000 7f8b40c00640 1 -- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b4414d8c0 msgr2=0x7f8b441e0d40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:46:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.371+0000 7f8b40c00640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b4414d8c0 0x7f8b441e0d40 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8b4406ca00 tx=0x7f8b300046c0 comp rx=0 tx=0).stop 2024-09-13T16:46:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.371+0000 7f8b49c00640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8b2406abb0 0x7f8b2406d070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c005c00 tx=0x7f8b3c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.378+0000 7f8b4bc00640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b44157c10 0x7f8b441e1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.378+0000 7f8b4bc00640 1 -- 172.21.15.28:0/3466524878 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b441ef7e0 con 0x7f8b44157c10 2024-09-13T16:46:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.378+0000 7f8b43e00640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b44157c10 0x7f8b441e1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:38.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.378+0000 7f8b43e00640 1 -- 172.21.15.28:0/3466524878 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8b440c3310 con 0x7f8b44157c10 2024-09-13T16:46:38.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.379+0000 7f8b43e00640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b44157c10 0x7f8b441e1280 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8b300097a0 tx=0x7f8b30009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:38.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.379+0000 7f8b40c00640 1 -- 172.21.15.28:0/3466524878 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30047960 con 0x7f8b44157c10 2024-09-13T16:46:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.379+0000 7f8b40c00640 1 -- 172.21.15.28:0/3466524878 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+5087 (secure 0 0 0) 0x7f8b3004b070 con 0x7f8b44157c10 2024-09-13T16:46:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.379+0000 7f8b40c00640 1 -- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b44157c10 msgr2=0x7f8b441e1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.379+0000 7f8b40c00640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b44157c10 0x7f8b441e1280 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8b300097a0 tx=0x7f8b30009190 comp rx=0 tx=0).stop 2024-09-13T16:46:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.379+0000 7f8b40c00640 1 -- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b44157c10 msgr2=0x7f8b441e1280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:46:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.379+0000 7f8b40c00640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b44157c10 0x7f8b441e1280 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8b300097a0 tx=0x7f8b30009190 comp rx=0 tx=0).stop 2024-09-13T16:46:38.381 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:46:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:46:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:46:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:46:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 18, 2024-09-13T16:46:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:46:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:46:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:46:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 0, 2024-09-13T16:46:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:46:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:46:38.382 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:46:38.383 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:38.383 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:38.383 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:38.383 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:38.383 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:38.383 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:38.383 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:38.383 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:38.383 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:38.383 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:38.383 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:38.384 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:46:38.384 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:46:38.384 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:38.384 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:38.384 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:38.384 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:38.384 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:38.384 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:38.384 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:38.384 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:38.384 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:38.384 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:38.384 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:38.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.385 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:46:38.385 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:46:38.385 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:46:38.385 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:46:38.385 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:46:38.385 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:46:38.385 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:46:38.385 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:46:38.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:46:38.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:46:38.385 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:46:38.385 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:46:38.385 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:46:38.386 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:46:38.386 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:46:38.386 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:46:38.386 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:46:38.386 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:46:38.386 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:46:38.386 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:46:38.386 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:46:38.386 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:46:38.386 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:46:38.386 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:46:38.386 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:46:38.386 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:46:38.386 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:46:38.387 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:38.387 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:46:38.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.387 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:46:38.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.387 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:46:38.387 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:46:38.387 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:38.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:38.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:38.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:46:38.387 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:38.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:38.388 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:46:38.388 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:38.388 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:38.388 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:38.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.388 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:46:38.388 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:46:38.388 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:38.388 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:38.388 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:38.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.389 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:46:38.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:46:38.389 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:38.389 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:38.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:38.389 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:46:38.389 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:38.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:38.389 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:46:38.390 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:38.390 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:38.390 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:38.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.390 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:46:38.390 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:46:38.390 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:38.390 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:38.390 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:38.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.390 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:46:38.390 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:46:38.390 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:46:38.391 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:46:38.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:46:38.391 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:46:38.391 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:38.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:46:38.391 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:46:38.391 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:46:38.391 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:38.391 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:38.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.391 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:46:38.391 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:46:38.392 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:46:38.392 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:46:38.392 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:46:38.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:38.392 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:38.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.392 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:46:38.392 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:46:38.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.392 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:38.392 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:38.392 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:46:38.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:38.392 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:38.393 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:46:38.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.393 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:38.393 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:38.393 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 5 2024-09-13T16:46:38.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:38.393 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:38.393 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:46:38.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.393 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:38.393 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:38.393 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 6 2024-09-13T16:46:38.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:38.394 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:38.394 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:46:38.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.394 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x2f018fb87aa4aafe", 2024-09-13T16:46:38.394 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "luminous", 2024-09-13T16:46:38.394 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:46:38.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.394 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:38.394 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:38.394 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4 2024-09-13T16:46:38.394 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:38.394 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:46:38.394 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": [ 2024-09-13T16:46:38.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:46:38.395 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:46:38.395 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:46:38.395 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:46:38.395 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:46:38.395 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:46:38.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:46:38.395 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:46:38.395 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:46:38.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.380+0000 7f8b4bc00640 1 -- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8b2406abb0 msgr2=0x7f8b2406d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:38.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.380+0000 7f8b4bc00640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8b2406abb0 0x7f8b2406d070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c005c00 tx=0x7f8b3c009040 comp rx=0 tx=0).stop 2024-09-13T16:46:38.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.380+0000 7f8b4bc00640 1 -- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b4406dcb0 msgr2=0x7f8b441e2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:38.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.380+0000 7f8b4bc00640 1 --2- 172.21.15.28:0/3466524878 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b4406dcb0 0x7f8b441e2600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8b3800b580 tx=0x7f8b3800ba40 comp rx=0 tx=0).stop 2024-09-13T16:46:38.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.380+0000 7f8b49c00640 1 -- 172.21.15.28:0/3466524878 reap_dead start 2024-09-13T16:46:38.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.380+0000 7f8b49c00640 1 -- 172.21.15.28:0/3466524878 reap_dead start 2024-09-13T16:46:38.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.380+0000 7f8b4bc00640 1 -- 172.21.15.28:0/3466524878 shutdown_connections 2024-09-13T16:46:38.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.380+0000 7f8b4bc00640 1 -- 172.21.15.28:0/3466524878 >> 172.21.15.28:0/3466524878 conn(0x7f8b440ca6b0 msgr2=0x7f8b4414e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:38.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.380+0000 7f8b4bc00640 1 -- 172.21.15.28:0/3466524878 shutdown_connections 2024-09-13T16:46:38.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:38.380+0000 7f8b4bc00640 1 -- 172.21.15.28:0/3466524878 wait complete. 2024-09-13T16:46:38.397 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-13T16:46:38.397 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-13T16:46:58.398 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-13T16:46:58.398 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-13T16:46:58.399 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-13T16:46:58.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-13T16:46:58.403 INFO:tasks.ceph.mon.a:Started 2024-09-13T16:46:58.404 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T16:46:58.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:46:58.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.683+0000 7f46d7e00640 1 Processor -- start 2024-09-13T16:46:58.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.684+0000 7f46d7e00640 1 -- start start 2024-09-13T16:46:58.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.685+0000 7f46d7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46d01570a0 0x7f46d0070040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:58.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.685+0000 7f46d7e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d015df30 con 0x7f46d0157480 2024-09-13T16:46:58.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.685+0000 7f46d7e00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d0161340 con 0x7f46d00c6270 2024-09-13T16:46:58.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.685+0000 7f46d7e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d015d320 con 0x7f46d01570a0 2024-09-13T16:46:58.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.685+0000 7f46d5400640 1 -- >> v2:172.21.15.28:3300/0 conn(0x7f46d01570a0 msgr2=0x7f46d0070040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:46:58.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.685+0000 7f46d5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46d01570a0 0x7f46d0070040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:58.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.685+0000 7f46d5e00640 1 -- >> v1:172.21.15.28:6789/0 conn(0x7f46d0157480 legacy=0x7f46d006d430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.28:6789/0 2024-09-13T16:46:58.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.685+0000 7f46d4a00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f46d00c6270 0x7f46d00c6ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:49902/0 (socket says 172.21.15.28:49902) 2024-09-13T16:46:58.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.685+0000 7f46d4a00640 1 -- 172.21.15.28:0/2902894520 learned_addr learned my addr 172.21.15.28:0/2902894520 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:46:58.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.686+0000 7f46cfe00640 1 -- 172.21.15.28:0/2902894520 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091718989 0 0) 0x7f46d0161340 con 0x7f46d00c6270 2024-09-13T16:46:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.686+0000 7f46cfe00640 1 -- 172.21.15.28:0/2902894520 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46b8003520 con 0x7f46d00c6270 2024-09-13T16:46:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.686+0000 7f46cfe00640 1 -- 172.21.15.28:0/2902894520 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1025997308 0 0) 0x7f46b8003520 con 0x7f46d00c6270 2024-09-13T16:46:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.686+0000 7f46cfe00640 1 -- 172.21.15.28:0/2902894520 >> v2:172.21.15.28:3300/0 conn(0x7f46d01570a0 msgr2=0x7f46d0070040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:46:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.686+0000 7f46cfe00640 1 --2- 172.21.15.28:0/2902894520 >> v2:172.21.15.28:3300/0 conn(0x7f46d01570a0 0x7f46d0070040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.686+0000 7f46cfe00640 1 -- 172.21.15.28:0/2902894520 >> v1:172.21.15.28:6789/0 conn(0x7f46d0157480 legacy=0x7f46d006d430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:46:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.686+0000 7f46cfe00640 1 -- 172.21.15.28:0/2902894520 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d015d600 con 0x7f46d00c6270 2024-09-13T16:46:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.686+0000 7f46cfe00640 1 -- 172.21.15.28:0/2902894520 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1654382301 0 0) 0x7f46bc001670 con 0x7f46d00c6270 2024-09-13T16:46:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.686+0000 7f46cfe00640 1 -- 172.21.15.28:0/2902894520 >> v1:172.21.15.191:6789/0 conn(0x7f46d00c6270 legacy=0x7f46d00c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.686+0000 7f46cfe00640 1 --2- 172.21.15.28:0/2902894520 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46b8003970 0x7f46b8003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.686+0000 7f46cfe00640 1 --2- 172.21.15.28:0/2902894520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b80047c0 0x7f46b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.686+0000 7f46cfe00640 1 --2- 172.21.15.28:0/2902894520 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46b80051f0 0x7f46b800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:58.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.686+0000 7f46cfe00640 1 -- 172.21.15.28:0/2902894520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d015d320 con 0x7f46b80047c0 2024-09-13T16:46:58.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.686+0000 7f46cfe00640 1 -- 172.21.15.28:0/2902894520 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b800c150 con 0x7f46b8003970 2024-09-13T16:46:58.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.686+0000 7f46cfe00640 1 -- 172.21.15.28:0/2902894520 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b800c310 con 0x7f46b80051f0 2024-09-13T16:46:58.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.687+0000 7f46d4a00640 1 -- 172.21.15.28:0/2902894520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b80047c0 msgr2=0x7f46b8004c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:46:58.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.687+0000 7f46d5e00640 1 --2- 172.21.15.28:0/2902894520 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46b80051f0 0x7f46b800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:58.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.687+0000 7f46d4a00640 1 --2- 172.21.15.28:0/2902894520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b80047c0 0x7f46b8004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:58.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.687+0000 7f46d5400640 1 --2- 172.21.15.28:0/2902894520 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46b8003970 0x7f46b8003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:58.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.687+0000 7f46d5e00640 1 -- 172.21.15.28:0/2902894520 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46b8003970 msgr2=0x7f46b8003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:46:58.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.687+0000 7f46d5e00640 1 --2- 172.21.15.28:0/2902894520 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46b8003970 0x7f46b8003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:58.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.687+0000 7f46d5e00640 1 -- 172.21.15.28:0/2902894520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b80047c0 msgr2=0x7f46b8004c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:46:58.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.687+0000 7f46d5e00640 1 --2- 172.21.15.28:0/2902894520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46b80047c0 0x7f46b8004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:58.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.687+0000 7f46d5e00640 1 -- 172.21.15.28:0/2902894520 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d015d600 con 0x7f46b80051f0 2024-09-13T16:46:58.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.689+0000 7f46d5e00640 1 -- 172.21.15.28:0/2902894520 reap_dead start 2024-09-13T16:46:58.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.689+0000 7f46d5400640 1 --2- 172.21.15.28:0/2902894520 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46b8003970 0x7f46b8003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:46:58.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.689+0000 7f46d5e00640 1 --2- 172.21.15.28:0/2902894520 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46b80051f0 0x7f46b800b980 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f46c8009e30 tx=0x7f46b80057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a917bcbc06737cfb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:58.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.689+0000 7f46cfe00640 1 -- 172.21.15.28:0/2902894520 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c8033070 con 0x7f46b80051f0 2024-09-13T16:46:58.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.689+0000 7f46cfe00640 1 -- 172.21.15.28:0/2902894520 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c80050d0 con 0x7f46b80051f0 2024-09-13T16:46:58.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.689+0000 7f46cfe00640 1 -- 172.21.15.28:0/2902894520 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c8005480 con 0x7f46b80051f0 2024-09-13T16:46:58.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.690+0000 7f46d7e00640 1 -- 172.21.15.28:0/2902894520 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46b80051f0 msgr2=0x7f46b800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:58.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.690+0000 7f46d7e00640 1 --2- 172.21.15.28:0/2902894520 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46b80051f0 0x7f46b800b980 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f46c8009e30 tx=0x7f46b80057f0 comp rx=0 tx=0).stop 2024-09-13T16:46:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.693+0000 7f46d7e00640 1 -- 172.21.15.28:0/2902894520 shutdown_connections 2024-09-13T16:46:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.693+0000 7f46d7e00640 1 --2- 172.21.15.28:0/2902894520 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46b80051f0 0x7f46b800b980 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.693+0000 7f46d7e00640 1 -- 172.21.15.28:0/2902894520 >> 172.21.15.28:0/2902894520 conn(0x7f46d0069e80 msgr2=0x7f46d006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:58.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.694+0000 7f46d7e00640 1 -- 172.21.15.28:0/2902894520 shutdown_connections 2024-09-13T16:46:58.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.694+0000 7f46d7e00640 1 -- 172.21.15.28:0/2902894520 wait complete. 2024-09-13T16:46:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.697+0000 7f46d7e00640 1 Processor -- start 2024-09-13T16:46:58.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.699+0000 7f46d7e00640 1 -- start start 2024-09-13T16:46:58.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.701+0000 7f46d7e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46b80051f0 0x7f46d01f22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.701+0000 7f46d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46d015a410 0x7f46d01f58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.701+0000 7f46d5400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46b80051f0 0x7f46d01f22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.701+0000 7f46d5400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46b80051f0 0x7f46d01f22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:37896/0 (socket says 172.21.15.28:37896) 2024-09-13T16:46:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.702+0000 7f46d4a00640 1 -- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46d015a410 msgr2=0x7f46d01f58c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:46:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.702+0000 7f46d4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46d015a410 0x7f46d01f58c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:46:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.702+0000 7f46d5400640 1 -- 172.21.15.28:0/1607856436 learned_addr learned my addr 172.21.15.28:0/1607856436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:46:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.702+0000 7f46d7e00640 1 --2- 172.21.15.28:0/1607856436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46d015ee20 0x7f46d015f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.702+0000 7f46d7e00640 1 -- 172.21.15.28:0/1607856436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d01f6170 con 0x7f46d015a410 2024-09-13T16:46:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.702+0000 7f46d7e00640 1 -- 172.21.15.28:0/1607856436 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d01f6450 con 0x7f46d015ee20 2024-09-13T16:46:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.702+0000 7f46d7e00640 1 -- 172.21.15.28:0/1607856436 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d01f6730 con 0x7f46b80051f0 2024-09-13T16:46:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.702+0000 7f46d5e00640 1 --2- 172.21.15.28:0/1607856436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46d015ee20 0x7f46d015f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:58.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.703+0000 7f46d5e00640 1 -- 172.21.15.28:0/1607856436 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46b80051f0 msgr2=0x7f46d01f22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.703+0000 7f46d5e00640 1 --2- 172.21.15.28:0/1607856436 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46b80051f0 0x7f46d01f22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.703+0000 7f46d5e00640 1 -- 172.21.15.28:0/1607856436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46d015a410 msgr2=0x7f46d01f58c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:46:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.703+0000 7f46d5e00640 1 --2- 172.21.15.28:0/1607856436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46d015a410 0x7f46d01f58c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.704+0000 7f46d5e00640 1 -- 172.21.15.28:0/1607856436 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d015e340 con 0x7f46d015ee20 2024-09-13T16:46:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.704+0000 7f46d5400640 1 --2- 172.21.15.28:0/1607856436 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46b80051f0 0x7f46d01f22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:46:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.704+0000 7f46d5e00640 1 --2- 172.21.15.28:0/1607856436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46d015ee20 0x7f46d015f2d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46d0157860 tx=0x7f46c8005cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.705+0000 7f46cd600640 1 -- 172.21.15.28:0/1607856436 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c8033070 con 0x7f46d015ee20 2024-09-13T16:46:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.705+0000 7f46cd600640 1 -- 172.21.15.28:0/1607856436 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c80396f0 con 0x7f46d015ee20 2024-09-13T16:46:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.705+0000 7f46cd600640 1 -- 172.21.15.28:0/1607856436 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c8039a80 con 0x7f46d015ee20 2024-09-13T16:46:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.705+0000 7f46d7e00640 1 -- 172.21.15.28:0/1607856436 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d015e620 con 0x7f46d015ee20 2024-09-13T16:46:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.705+0000 7f46d7e00640 1 -- 172.21.15.28:0/1607856436 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d01f5e00 con 0x7f46d015ee20 2024-09-13T16:46:58.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.706+0000 7f46d7e00640 1 -- 172.21.15.28:0/1607856436 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d00711c0 con 0x7f46d015ee20 2024-09-13T16:46:58.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.710+0000 7f46cd600640 1 -- 172.21.15.28:0/1607856436 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f46c8039be0 con 0x7f46d015ee20 2024-09-13T16:46:58.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.712+0000 7f46cd600640 1 --2- 172.21.15.28:0/1607856436 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f46b006abe0 0x7f46b006d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:46:58.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.712+0000 7f46cd600640 1 -- 172.21.15.28:0/1607856436 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f46c8008050 con 0x7f46d015ee20 2024-09-13T16:46:58.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.712+0000 7f46d5400640 1 --2- 172.21.15.28:0/1607856436 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f46b006abe0 0x7f46b006d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:46:58.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.713+0000 7f46d5400640 1 --2- 172.21.15.28:0/1607856436 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f46b006abe0 0x7f46b006d0a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f46c40094d0 tx=0x7f46c40090e0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:46:58.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:58.714+0000 7f46cd600640 1 -- 172.21.15.28:0/1607856436 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f46c8074b80 con 0x7f46d015ee20 2024-09-13T16:46:59.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:59.105+0000 7f46d7e00640 1 -- 172.21.15.28:0/1607856436 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f46d00c6890 con 0x7f46d015ee20 2024-09-13T16:46:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:59.105+0000 7f46cd600640 1 -- 172.21.15.28:0/1607856436 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7f46c803d080 con 0x7f46d015ee20 2024-09-13T16:46:59.107 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":18,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:46:59.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:59.108+0000 7f46d7e00640 1 -- 172.21.15.28:0/1607856436 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f46b006abe0 msgr2=0x7f46b006d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:59.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:59.108+0000 7f46d7e00640 1 --2- 172.21.15.28:0/1607856436 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f46b006abe0 0x7f46b006d0a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f46c40094d0 tx=0x7f46c40090e0 comp rx=0 tx=0).stop 2024-09-13T16:46:59.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:59.108+0000 7f46d7e00640 1 -- 172.21.15.28:0/1607856436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46d015ee20 msgr2=0x7f46d015f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:46:59.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:59.108+0000 7f46d7e00640 1 --2- 172.21.15.28:0/1607856436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46d015ee20 0x7f46d015f2d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46d0157860 tx=0x7f46c8005cc0 comp rx=0 tx=0).stop 2024-09-13T16:46:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:59.110+0000 7f46d7e00640 1 -- 172.21.15.28:0/1607856436 shutdown_connections 2024-09-13T16:46:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:59.110+0000 7f46d7e00640 1 --2- 172.21.15.28:0/1607856436 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f46b006abe0 0x7f46b006d0a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:59.110+0000 7f46d7e00640 1 --2- 172.21.15.28:0/1607856436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46d015ee20 0x7f46d015f2d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:59.110+0000 7f46d7e00640 1 --2- 172.21.15.28:0/1607856436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46d015a410 0x7f46d01f58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:59.110+0000 7f46d7e00640 1 --2- 172.21.15.28:0/1607856436 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46b80051f0 0x7f46d01f22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:46:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:59.110+0000 7f46d7e00640 1 -- 172.21.15.28:0/1607856436 >> 172.21.15.28:0/1607856436 conn(0x7f46d0069e80 msgr2=0x7f46d015ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:46:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:59.110+0000 7f46d7e00640 1 -- 172.21.15.28:0/1607856436 shutdown_connections 2024-09-13T16:46:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:46:59.111+0000 7f46d7e00640 1 -- 172.21.15.28:0/1607856436 wait complete. 2024-09-13T16:47:02.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:47:02.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.495+0000 7fde11e00640 1 Processor -- start 2024-09-13T16:47:02.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.495+0000 7fde11e00640 1 -- start start 2024-09-13T16:47:02.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.501+0000 7fde11e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fde0c0ca590 0x7fde0c0c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:02.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.501+0000 7fde11e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fde0c0c9840 0x7fde0c06af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:02.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.501+0000 7fde11e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0c0c9c70 con 0x7fde0c152e20 2024-09-13T16:47:02.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.501+0000 7fde11e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde0c159be0 con 0x7fde0c0ca590 2024-09-13T16:47:02.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.501+0000 7fde11e00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde0c158d80 con 0x7fde0c0c9840 2024-09-13T16:47:02.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.502+0000 7fde0b400640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fde0c0c9840 0x7fde0c06af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:02.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.502+0000 7fde0aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fde0c152e20 0x7fde0c0caf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43392/0 (socket says 172.21.15.28:43392) 2024-09-13T16:47:02.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.502+0000 7fde0aa00640 1 -- 172.21.15.28:0/2745214439 learned_addr learned my addr 172.21.15.28:0/2745214439 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:47:02.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.502+0000 7fde0a000640 1 --2- 172.21.15.28:0/2745214439 >> v2:172.21.15.28:3300/0 conn(0x7fde0c0ca590 0x7fde0c0c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:02.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.502+0000 7fde0a000640 1 -- 172.21.15.28:0/2745214439 >> v2:172.21.15.28:3300/0 conn(0x7fde0c0ca590 msgr2=0x7fde0c0c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T16:47:02.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.502+0000 7fde0a000640 1 -- 172.21.15.28:0/2745214439 >> v2:172.21.15.28:3300/0 conn(0x7fde0c0ca590 msgr2=0x7fde0c0c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:47:02.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.502+0000 7fde0a000640 1 --2- 172.21.15.28:0/2745214439 >> v2:172.21.15.28:3300/0 conn(0x7fde0c0ca590 0x7fde0c0c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:47:02.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.502+0000 7fde0a000640 1 --2- 172.21.15.28:0/2745214439 >> v2:172.21.15.28:3300/0 conn(0x7fde0c0ca590 0x7fde0c0c9300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:02.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.503+0000 7fde0b400640 1 -- 172.21.15.28:0/2745214439 >> v2:172.21.15.28:3300/0 conn(0x7fde0c0ca590 msgr2=0x7fde0c0c9300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:02.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.503+0000 7fde0b400640 1 --2- 172.21.15.28:0/2745214439 >> v2:172.21.15.28:3300/0 conn(0x7fde0c0ca590 0x7fde0c0c9300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:02.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.503+0000 7fde0b400640 1 -- 172.21.15.28:0/2745214439 >> v1:172.21.15.28:6789/0 conn(0x7fde0c152e20 legacy=0x7fde0c0caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:02.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.503+0000 7fde0b400640 1 -- 172.21.15.28:0/2745214439 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c159060 con 0x7fde0c0c9840 2024-09-13T16:47:02.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.505+0000 7fde0b400640 1 --2- 172.21.15.28:0/2745214439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde0c0c9840 0x7fde0c06af90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fde0c0c9020 tx=0x7fde0402a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab4c3567835e2351 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:02.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.505+0000 7fde09600640 1 -- 172.21.15.28:0/2745214439 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04007740 con 0x7fde0c0c9840 2024-09-13T16:47:02.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.505+0000 7fde09600640 1 -- 172.21.15.28:0/2745214439 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0402d050 con 0x7fde0c0c9840 2024-09-13T16:47:02.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.505+0000 7fde09600640 1 -- 172.21.15.28:0/2745214439 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04007ac0 con 0x7fde0c0c9840 2024-09-13T16:47:02.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.506+0000 7fde11e00640 1 -- 172.21.15.28:0/2745214439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde0c0c9840 msgr2=0x7fde0c06af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:02.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.506+0000 7fde11e00640 1 --2- 172.21.15.28:0/2745214439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde0c0c9840 0x7fde0c06af90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fde0c0c9020 tx=0x7fde0402a710 comp rx=0 tx=0).stop 2024-09-13T16:47:02.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.506+0000 7fde11e00640 1 -- 172.21.15.28:0/2745214439 shutdown_connections 2024-09-13T16:47:02.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.506+0000 7fde11e00640 1 --2- 172.21.15.28:0/2745214439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde0c0c9840 0x7fde0c06af90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:02.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.506+0000 7fde11e00640 1 --2- 172.21.15.28:0/2745214439 >> v2:172.21.15.28:3300/0 conn(0x7fde0c0ca590 0x7fde0c0c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:02.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.506+0000 7fde11e00640 1 -- 172.21.15.28:0/2745214439 >> 172.21.15.28:0/2745214439 conn(0x7fde0c065c50 msgr2=0x7fde0c066060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:02.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.506+0000 7fde11e00640 1 -- 172.21.15.28:0/2745214439 shutdown_connections 2024-09-13T16:47:02.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.507+0000 7fde11e00640 1 -- 172.21.15.28:0/2745214439 wait complete. 2024-09-13T16:47:02.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.509+0000 7fde11e00640 1 Processor -- start 2024-09-13T16:47:02.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.509+0000 7fde11e00640 1 -- start start 2024-09-13T16:47:02.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.512+0000 7fde11e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde0c0c9840 0x7fde0c1ef180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:02.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.512+0000 7fde11e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde0c0ca590 0x7fde0c148970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:02.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.512+0000 7fde11e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde0c152e20 0x7fde0c1dfd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:02.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.512+0000 7fde11e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c0c8420 con 0x7fde0c0ca590 2024-09-13T16:47:02.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.512+0000 7fde11e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c1e09b0 con 0x7fde0c0c9840 2024-09-13T16:47:02.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.512+0000 7fde11e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c1ef6c0 con 0x7fde0c152e20 2024-09-13T16:47:02.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.512+0000 7fde0aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde0c0c9840 0x7fde0c1ef180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:02.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.512+0000 7fde0aa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde0c0c9840 0x7fde0c1ef180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:47826/0 (socket says 172.21.15.28:47826) 2024-09-13T16:47:02.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.512+0000 7fde0aa00640 1 -- 172.21.15.28:0/351847474 learned_addr learned my addr 172.21.15.28:0/351847474 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:02.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.513+0000 7fde0aa00640 1 -- 172.21.15.28:0/351847474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde0c152e20 msgr2=0x7fde0c1dfd60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:47:02.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.513+0000 7fde0aa00640 1 --2- 172.21.15.28:0/351847474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde0c152e20 0x7fde0c1dfd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:02.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.513+0000 7fde0aa00640 1 -- 172.21.15.28:0/351847474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde0c0ca590 msgr2=0x7fde0c148970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:47:02.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.513+0000 7fde0a000640 1 --2- 172.21.15.28:0/351847474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde0c0ca590 0x7fde0c148970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:02.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.513+0000 7fde0aa00640 1 --2- 172.21.15.28:0/351847474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde0c0ca590 0x7fde0c148970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:02.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.513+0000 7fde0aa00640 1 -- 172.21.15.28:0/351847474 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde04009a70 con 0x7fde0c0c9840 2024-09-13T16:47:02.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.513+0000 7fde0a000640 1 --2- 172.21.15.28:0/351847474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde0c0ca590 0x7fde0c148970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:47:02.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.514+0000 7fde0aa00640 1 --2- 172.21.15.28:0/351847474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde0c0c9840 0x7fde0c1ef180 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fddfc00c340 tx=0x7fddfc00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:02.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.514+0000 7fde03400640 1 -- 172.21.15.28:0/351847474 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc005ce0 con 0x7fde0c0c9840 2024-09-13T16:47:02.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.514+0000 7fde03400640 1 -- 172.21.15.28:0/351847474 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddfc005e40 con 0x7fde0c0c9840 2024-09-13T16:47:02.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.514+0000 7fde03400640 1 -- 172.21.15.28:0/351847474 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0072f0 con 0x7fde0c0c9840 2024-09-13T16:47:02.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.514+0000 7fde11e00640 1 -- 172.21.15.28:0/351847474 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0c0c50b0 con 0x7fde0c0c9840 2024-09-13T16:47:02.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.515+0000 7fde11e00640 1 -- 172.21.15.28:0/351847474 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde0c0c4dd0 con 0x7fde0c0c9840 2024-09-13T16:47:02.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.516+0000 7fde11e00640 1 -- 172.21.15.28:0/351847474 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0c0c9300 con 0x7fde0c0c9840 2024-09-13T16:47:02.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.520+0000 7fde03400640 1 -- 172.21.15.28:0/351847474 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fddfc007510 con 0x7fde0c0c9840 2024-09-13T16:47:02.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.521+0000 7fde03400640 1 --2- 172.21.15.28:0/351847474 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fddec06ac00 0x7fddec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:02.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.521+0000 7fde03400640 1 -- 172.21.15.28:0/351847474 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fddfc088a00 con 0x7fde0c0c9840 2024-09-13T16:47:02.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.522+0000 7fde0a000640 1 --2- 172.21.15.28:0/351847474 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fddec06ac00 0x7fddec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:02.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.523+0000 7fde0a000640 1 --2- 172.21.15.28:0/351847474 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fddec06ac00 0x7fddec06d0c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fde0c14ce40 tx=0x7fddf8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:02.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.525+0000 7fde03400640 1 -- 172.21.15.28:0/351847474 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fddfc053b40 con 0x7fde0c0c9840 2024-09-13T16:47:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.975+0000 7fde11e00640 1 -- 172.21.15.28:0/351847474 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fde0c0c9c70 con 0x7fde0c0c9840 2024-09-13T16:47:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.976+0000 7fde03400640 1 -- 172.21.15.28:0/351847474 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7fde0c0c9c70 con 0x7fde0c0c9840 2024-09-13T16:47:02.977 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":18,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":25,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:47:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.979+0000 7fde11e00640 1 -- 172.21.15.28:0/351847474 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fddec06ac00 msgr2=0x7fddec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.979+0000 7fde11e00640 1 --2- 172.21.15.28:0/351847474 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fddec06ac00 0x7fddec06d0c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fde0c14ce40 tx=0x7fddf8008040 comp rx=0 tx=0).stop 2024-09-13T16:47:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.979+0000 7fde11e00640 1 -- 172.21.15.28:0/351847474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde0c0c9840 msgr2=0x7fde0c1ef180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.979+0000 7fde11e00640 1 --2- 172.21.15.28:0/351847474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde0c0c9840 0x7fde0c1ef180 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fddfc00c340 tx=0x7fddfc00c800 comp rx=0 tx=0).stop 2024-09-13T16:47:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.979+0000 7fde11e00640 1 -- 172.21.15.28:0/351847474 shutdown_connections 2024-09-13T16:47:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.979+0000 7fde11e00640 1 --2- 172.21.15.28:0/351847474 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fddec06ac00 0x7fddec06d0c0 secure :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fde0c14ce40 tx=0x7fddf8008040 comp rx=0 tx=0).stop 2024-09-13T16:47:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.979+0000 7fde11e00640 1 --2- 172.21.15.28:0/351847474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde0c152e20 0x7fde0c1dfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.979+0000 7fde11e00640 1 --2- 172.21.15.28:0/351847474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde0c0ca590 0x7fde0c148970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.979+0000 7fde11e00640 1 --2- 172.21.15.28:0/351847474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde0c0c9840 0x7fde0c1ef180 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.979+0000 7fde11e00640 1 -- 172.21.15.28:0/351847474 >> 172.21.15.28:0/351847474 conn(0x7fde0c065c50 msgr2=0x7fde0c150ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.979+0000 7fde11e00640 1 -- 172.21.15.28:0/351847474 shutdown_connections 2024-09-13T16:47:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:02.979+0000 7fde11e00640 1 -- 172.21.15.28:0/351847474 wait complete. 2024-09-13T16:47:04.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:47:04.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:47:06.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:47:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.332+0000 7fd270800640 1 Processor -- start 2024-09-13T16:47:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.332+0000 7fd270800640 1 -- start start 2024-09-13T16:47:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.334+0000 7fd270800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd268150a70 0x7fd2680c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.334+0000 7fd270800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2680c6720 0x7fd2680c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.334+0000 7fd270800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd2680c53a0 0x7fd268156c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.334+0000 7fd270800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2680c64f0 con 0x7fd268150a70 2024-09-13T16:47:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.334+0000 7fd270800640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd268159f70 con 0x7fd2680c53a0 2024-09-13T16:47:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.334+0000 7fd270800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2680c1ea0 con 0x7fd2680c6720 2024-09-13T16:47:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.334+0000 7fd26de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd268150a70 0x7fd2680c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.334+0000 7fd26de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd268150a70 0x7fd2680c1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53284/0 (socket says 172.21.15.28:53284) 2024-09-13T16:47:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.334+0000 7fd26de00640 1 -- 172.21.15.28:0/249390803 learned_addr learned my addr 172.21.15.28:0/249390803 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.334+0000 7fd26d400640 1 --2- 172.21.15.28:0/249390803 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2680c6720 0x7fd2680c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.334+0000 7fd26e800640 1 --2- 172.21.15.28:0/249390803 >> v2:172.21.15.191:3300/0 conn(0x7fd2680c53a0 0x7fd268156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.335+0000 7fd26d400640 1 -- 172.21.15.28:0/249390803 >> v2:172.21.15.191:3300/0 conn(0x7fd2680c53a0 msgr2=0x7fd268156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:47:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.335+0000 7fd26d400640 1 --2- 172.21.15.28:0/249390803 >> v2:172.21.15.191:3300/0 conn(0x7fd2680c53a0 0x7fd268156c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.335+0000 7fd26d400640 1 -- 172.21.15.28:0/249390803 >> v2:172.21.15.28:3300/0 conn(0x7fd268150a70 msgr2=0x7fd2680c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:47:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.335+0000 7fd26d400640 1 --2- 172.21.15.28:0/249390803 >> v2:172.21.15.28:3300/0 conn(0x7fd268150a70 0x7fd2680c1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.335+0000 7fd26d400640 1 -- 172.21.15.28:0/249390803 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26815a1e0 con 0x7fd2680c6720 2024-09-13T16:47:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.336+0000 7fd26d400640 1 --2- 172.21.15.28:0/249390803 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2680c6720 0x7fd2680c4e60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd268072580 tx=0x7fd25802a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a15bf24cb199c84 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.336+0000 7fd26ca00640 1 -- 172.21.15.28:0/249390803 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258032440 con 0x7fd2680c6720 2024-09-13T16:47:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.336+0000 7fd26ca00640 1 -- 172.21.15.28:0/249390803 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2580325a0 con 0x7fd2680c6720 2024-09-13T16:47:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.336+0000 7fd26ca00640 1 -- 172.21.15.28:0/249390803 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258032930 con 0x7fd2680c6720 2024-09-13T16:47:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.337+0000 7fd270800640 1 -- 172.21.15.28:0/249390803 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2680c6720 msgr2=0x7fd2680c4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.337+0000 7fd270800640 1 --2- 172.21.15.28:0/249390803 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2680c6720 0x7fd2680c4e60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd268072580 tx=0x7fd25802a1a0 comp rx=0 tx=0).stop 2024-09-13T16:47:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.337+0000 7fd270800640 1 -- 172.21.15.28:0/249390803 shutdown_connections 2024-09-13T16:47:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.337+0000 7fd270800640 1 --2- 172.21.15.28:0/249390803 >> v2:172.21.15.191:3300/0 conn(0x7fd2680c53a0 0x7fd268156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.337+0000 7fd270800640 1 --2- 172.21.15.28:0/249390803 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2680c6720 0x7fd2680c4e60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.337+0000 7fd270800640 1 --2- 172.21.15.28:0/249390803 >> v2:172.21.15.28:3300/0 conn(0x7fd268150a70 0x7fd2680c1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.337+0000 7fd270800640 1 -- 172.21.15.28:0/249390803 >> 172.21.15.28:0/249390803 conn(0x7fd268069e80 msgr2=0x7fd26806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.337+0000 7fd270800640 1 -- 172.21.15.28:0/249390803 shutdown_connections 2024-09-13T16:47:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.337+0000 7fd270800640 1 -- 172.21.15.28:0/249390803 wait complete. 2024-09-13T16:47:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.339+0000 7fd270800640 1 Processor -- start 2024-09-13T16:47:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.339+0000 7fd270800640 1 -- start start 2024-09-13T16:47:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.340+0000 7fd270800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2680c53a0 0x7fd2681e10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.340+0000 7fd270800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2680c6720 0x7fd268144060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.340+0000 7fd270800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd268150a70 0x7fd2681445a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.340+0000 7fd270800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd268143cf0 con 0x7fd268150a70 2024-09-13T16:47:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.340+0000 7fd270800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd268071c00 con 0x7fd2680c53a0 2024-09-13T16:47:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.340+0000 7fd270800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd268070d80 con 0x7fd2680c6720 2024-09-13T16:47:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.340+0000 7fd26d400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2680c6720 0x7fd268144060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.340+0000 7fd26d400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2680c6720 0x7fd268144060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:38570/0 (socket says 172.21.15.28:38570) 2024-09-13T16:47:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.340+0000 7fd26d400640 1 -- 172.21.15.28:0/622497349 learned_addr learned my addr 172.21.15.28:0/622497349 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.340+0000 7fd26e800640 1 --2- 172.21.15.28:0/622497349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd268150a70 0x7fd2681445a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.340+0000 7fd26de00640 1 --2- 172.21.15.28:0/622497349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2680c53a0 0x7fd2681e10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.341+0000 7fd26d400640 1 -- 172.21.15.28:0/622497349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2680c53a0 msgr2=0x7fd2681e10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.341+0000 7fd26d400640 1 --2- 172.21.15.28:0/622497349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2680c53a0 0x7fd2681e10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.341+0000 7fd26d400640 1 -- 172.21.15.28:0/622497349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd268150a70 msgr2=0x7fd2681445a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.341+0000 7fd26d400640 1 --2- 172.21.15.28:0/622497349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd268150a70 0x7fd2681445a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.341+0000 7fd26d400640 1 -- 172.21.15.28:0/622497349 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd258009400 con 0x7fd2680c6720 2024-09-13T16:47:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.341+0000 7fd26e800640 1 --2- 172.21.15.28:0/622497349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd268150a70 0x7fd2681445a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:47:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.342+0000 7fd26d400640 1 --2- 172.21.15.28:0/622497349 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2680c6720 0x7fd268144060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd258000c00 tx=0x7fd2580046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.342+0000 7fd262000640 1 -- 172.21.15.28:0/622497349 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25803d8f0 con 0x7fd2680c6720 2024-09-13T16:47:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.342+0000 7fd262000640 1 -- 172.21.15.28:0/622497349 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25803da50 con 0x7fd2680c6720 2024-09-13T16:47:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.342+0000 7fd270800640 1 -- 172.21.15.28:0/622497349 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd268070ff0 con 0x7fd2680c6720 2024-09-13T16:47:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.342+0000 7fd262000640 1 -- 172.21.15.28:0/622497349 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25803ddc0 con 0x7fd2680c6720 2024-09-13T16:47:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.343+0000 7fd270800640 1 -- 172.21.15.28:0/622497349 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2680715b0 con 0x7fd2680c6720 2024-09-13T16:47:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.345+0000 7fd262000640 1 -- 172.21.15.28:0/622497349 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd258039040 con 0x7fd2680c6720 2024-09-13T16:47:06.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.345+0000 7fd270800640 1 -- 172.21.15.28:0/622497349 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2680c4750 con 0x7fd2680c6720 2024-09-13T16:47:06.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.346+0000 7fd262000640 1 --2- 172.21.15.28:0/622497349 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd24c06ac00 0x7fd24c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:06.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.346+0000 7fd262000640 1 -- 172.21.15.28:0/622497349 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd2580a9fc0 con 0x7fd2680c6720 2024-09-13T16:47:06.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.346+0000 7fd26de00640 1 --2- 172.21.15.28:0/622497349 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd24c06ac00 0x7fd24c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:06.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.347+0000 7fd26de00640 1 --2- 172.21.15.28:0/622497349 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd24c06ac00 0x7fd24c06d0c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009ac0 tx=0x7fd25c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:06.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.352+0000 7fd262000640 1 -- 172.21.15.28:0/622497349 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd258075080 con 0x7fd2680c6720 2024-09-13T16:47:06.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.865+0000 7fd270800640 1 -- 172.21.15.28:0/622497349 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd26814de40 con 0x7fd2680c6720 2024-09-13T16:47:06.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.866+0000 7fd262000640 1 -- 172.21.15.28:0/622497349 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7fd258037620 con 0x7fd2680c6720 2024-09-13T16:47:06.871 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:47:06.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.870+0000 7fd270800640 1 -- 172.21.15.28:0/622497349 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd24c06ac00 msgr2=0x7fd24c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:06.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.870+0000 7fd270800640 1 --2- 172.21.15.28:0/622497349 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd24c06ac00 0x7fd24c06d0c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009ac0 tx=0x7fd25c008040 comp rx=0 tx=0).stop 2024-09-13T16:47:06.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.870+0000 7fd270800640 1 -- 172.21.15.28:0/622497349 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2680c6720 msgr2=0x7fd268144060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:06.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.870+0000 7fd270800640 1 --2- 172.21.15.28:0/622497349 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2680c6720 0x7fd268144060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd258000c00 tx=0x7fd2580046c0 comp rx=0 tx=0).stop 2024-09-13T16:47:06.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.871+0000 7fd270800640 1 -- 172.21.15.28:0/622497349 shutdown_connections 2024-09-13T16:47:06.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.871+0000 7fd270800640 1 --2- 172.21.15.28:0/622497349 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fd24c06ac00 0x7fd24c06d0c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:06.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.871+0000 7fd270800640 1 --2- 172.21.15.28:0/622497349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd268150a70 0x7fd2681445a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:06.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.871+0000 7fd270800640 1 --2- 172.21.15.28:0/622497349 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2680c6720 0x7fd268144060 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:06.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.871+0000 7fd270800640 1 --2- 172.21.15.28:0/622497349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2680c53a0 0x7fd2681e10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:06.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.871+0000 7fd270800640 1 -- 172.21.15.28:0/622497349 >> 172.21.15.28:0/622497349 conn(0x7fd268069e80 msgr2=0x7fd268146f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:06.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.872+0000 7fd270800640 1 -- 172.21.15.28:0/622497349 shutdown_connections 2024-09-13T16:47:06.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:06.872+0000 7fd270800640 1 -- 172.21.15.28:0/622497349 wait complete. 2024-09-13T16:47:06.894 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T16:47:06.894 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-13T16:47:06.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T16:47:07.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.158+0000 7fcf6b200640 1 Processor -- start 2024-09-13T16:47:07.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.158+0000 7fcf6b200640 1 -- start start 2024-09-13T16:47:07.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.159+0000 7fcf6b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcf6c0ccbc0 0x7fcf6c155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.159+0000 7fcf6b200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf6c06af90 0x7fcf6c156520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.160+0000 7fcf6b200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcf6c06ce80 0x7fcf6c158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.160+0000 7fcf6b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6c06c150 con 0x7fcf6c0ccbc0 2024-09-13T16:47:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.160+0000 7fcf6b200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6c06c570 con 0x7fcf6c06ce80 2024-09-13T16:47:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.160+0000 7fcf6b200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c15d0a0 con 0x7fcf6c06af90 2024-09-13T16:47:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.160+0000 7fcf6a800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcf6c06ce80 0x7fcf6c158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.160+0000 7fcf69400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf6c06af90 0x7fcf6c156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.160+0000 7fcf6a800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcf6c06ce80 0x7fcf6c158bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:47856/0 (socket says 172.21.15.28:47856) 2024-09-13T16:47:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.160+0000 7fcf6a800640 1 -- 172.21.15.28:0/3213238400 learned_addr learned my addr 172.21.15.28:0/3213238400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.160+0000 7fcf6a800640 1 -- 172.21.15.28:0/3213238400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf6c06af90 msgr2=0x7fcf6c156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:47:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.160+0000 7fcf6a800640 1 --2- 172.21.15.28:0/3213238400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf6c06af90 0x7fcf6c156520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.160+0000 7fcf6a800640 1 -- 172.21.15.28:0/3213238400 >> v2:172.21.15.28:3300/0 conn(0x7fcf6c0ccbc0 msgr2=0x7fcf6c155fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.161+0000 7fcf69e00640 1 --2- 172.21.15.28:0/3213238400 >> v2:172.21.15.28:3300/0 conn(0x7fcf6c0ccbc0 0x7fcf6c155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.161+0000 7fcf6a800640 1 --2- 172.21.15.28:0/3213238400 >> v2:172.21.15.28:3300/0 conn(0x7fcf6c0ccbc0 0x7fcf6c155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.161+0000 7fcf6a800640 1 -- 172.21.15.28:0/3213238400 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6c0c75c0 con 0x7fcf6c06ce80 2024-09-13T16:47:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.161+0000 7fcf69e00640 1 --2- 172.21.15.28:0/3213238400 >> v2:172.21.15.28:3300/0 conn(0x7fcf6c0ccbc0 0x7fcf6c155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:47:07.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.161+0000 7fcf69400640 1 --2- 172.21.15.28:0/3213238400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf6c06af90 0x7fcf6c156520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:47:07.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.162+0000 7fcf6a800640 1 --2- 172.21.15.28:0/3213238400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf6c06ce80 0x7fcf6c158bc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c0c1b00 tx=0x7fcf6402a6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4db4218fa37b386 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:07.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.162+0000 7fcf68a00640 1 -- 172.21.15.28:0/3213238400 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf64031600 con 0x7fcf6c06ce80 2024-09-13T16:47:07.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.163+0000 7fcf68a00640 1 -- 172.21.15.28:0/3213238400 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf6402d050 con 0x7fcf6c06ce80 2024-09-13T16:47:07.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.163+0000 7fcf68a00640 1 -- 172.21.15.28:0/3213238400 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf64031980 con 0x7fcf6c06ce80 2024-09-13T16:47:07.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.163+0000 7fcf6b200640 1 -- 172.21.15.28:0/3213238400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf6c06ce80 msgr2=0x7fcf6c158bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.163+0000 7fcf6b200640 1 --2- 172.21.15.28:0/3213238400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf6c06ce80 0x7fcf6c158bc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c0c1b00 tx=0x7fcf6402a6d0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.164+0000 7fcf6b200640 1 -- 172.21.15.28:0/3213238400 shutdown_connections 2024-09-13T16:47:07.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.164+0000 7fcf6b200640 1 --2- 172.21.15.28:0/3213238400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf6c06ce80 0x7fcf6c158bc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.164+0000 7fcf6b200640 1 --2- 172.21.15.28:0/3213238400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf6c06af90 0x7fcf6c156520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.164+0000 7fcf6b200640 1 --2- 172.21.15.28:0/3213238400 >> v2:172.21.15.28:3300/0 conn(0x7fcf6c0ccbc0 0x7fcf6c155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.164+0000 7fcf6b200640 1 -- 172.21.15.28:0/3213238400 >> 172.21.15.28:0/3213238400 conn(0x7fcf6c06a5a0 msgr2=0x7fcf6c06ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:07.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.164+0000 7fcf6b200640 1 -- 172.21.15.28:0/3213238400 shutdown_connections 2024-09-13T16:47:07.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.164+0000 7fcf6b200640 1 -- 172.21.15.28:0/3213238400 wait complete. 2024-09-13T16:47:07.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.165+0000 7fcf6b200640 1 Processor -- start 2024-09-13T16:47:07.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.166+0000 7fcf6b200640 1 -- start start 2024-09-13T16:47:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.167+0000 7fcf6b200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf6c06af90 0x7fcf6c157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.167+0000 7fcf6b200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf6c0ccbc0 0x7fcf6c1f5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.167+0000 7fcf6b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c0ccfa0 0x7fcf6c1f6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.167+0000 7fcf6b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c157960 con 0x7fcf6c0ccfa0 2024-09-13T16:47:07.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.167+0000 7fcf6b200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c153640 con 0x7fcf6c0ccbc0 2024-09-13T16:47:07.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.167+0000 7fcf6b200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c0c4070 con 0x7fcf6c06af90 2024-09-13T16:47:07.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.167+0000 7fcf69e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf6c06af90 0x7fcf6c157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.167+0000 7fcf69e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf6c06af90 0x7fcf6c157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:38574/0 (socket says 172.21.15.28:38574) 2024-09-13T16:47:07.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.167+0000 7fcf69e00640 1 -- 172.21.15.28:0/2549864584 learned_addr learned my addr 172.21.15.28:0/2549864584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:07.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.167+0000 7fcf69400640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf6c0ccbc0 0x7fcf6c1f5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.168+0000 7fcf6a800640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c0ccfa0 0x7fcf6c1f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.167+0000 7fcf69e00640 1 -- 172.21.15.28:0/2549864584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf6c0ccbc0 msgr2=0x7fcf6c1f5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.168+0000 7fcf69e00640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf6c0ccbc0 0x7fcf6c1f5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.168+0000 7fcf69e00640 1 -- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c0ccfa0 msgr2=0x7fcf6c1f6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.168+0000 7fcf69e00640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c0ccfa0 0x7fcf6c1f6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.168+0000 7fcf69e00640 1 -- 172.21.15.28:0/2549864584 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf58009400 con 0x7fcf6c06af90 2024-09-13T16:47:07.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.169+0000 7fcf6a800640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c0ccfa0 0x7fcf6c1f6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:47:07.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.169+0000 7fcf69400640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcf6c0ccbc0 0x7fcf6c1f5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:47:07.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.169+0000 7fcf69e00640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf6c06af90 0x7fcf6c157420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00c820 tx=0x7fcf5c00cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:07.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.169+0000 7fcf62000640 1 -- 172.21.15.28:0/2549864584 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c007a30 con 0x7fcf6c06af90 2024-09-13T16:47:07.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.170+0000 7fcf62000640 1 -- 172.21.15.28:0/2549864584 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf5c007b90 con 0x7fcf6c06af90 2024-09-13T16:47:07.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.170+0000 7fcf62000640 1 -- 172.21.15.28:0/2549864584 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c00ce10 con 0x7fcf6c06af90 2024-09-13T16:47:07.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.170+0000 7fcf6b200640 1 -- 172.21.15.28:0/2549864584 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf64009b30 con 0x7fcf6c06af90 2024-09-13T16:47:07.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.170+0000 7fcf6b200640 1 -- 172.21.15.28:0/2549864584 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf6c0b2260 con 0x7fcf6c06af90 2024-09-13T16:47:07.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.171+0000 7fcf6b200640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c06ce80 0x7fcf6c0c3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.172+0000 7fcf6b200640 1 -- 172.21.15.28:0/2549864584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c06ab50 con 0x7fcf6c06ce80 2024-09-13T16:47:07.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.172+0000 7fcf69400640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c06ce80 0x7fcf6c0c3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.172+0000 7fcf69400640 1 -- 172.21.15.28:0/2549864584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf6c06bad0 con 0x7fcf6c06ce80 2024-09-13T16:47:07.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.173+0000 7fcf69400640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c06ce80 0x7fcf6c0c3a10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c1ea5f0 tx=0x7fcf58010c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:07.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.173+0000 7fcf62000640 1 -- 172.21.15.28:0/2549864584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf58010ee0 con 0x7fcf6c06ce80 2024-09-13T16:47:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.174+0000 7fcf62000640 1 -- 172.21.15.28:0/2549864584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fcf5800eba0 con 0x7fcf6c06ce80 2024-09-13T16:47:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.174+0000 7fcf62000640 1 -- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c06ce80 msgr2=0x7fcf6c0c3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.174+0000 7fcf62000640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c06ce80 0x7fcf6c0c3a10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c1ea5f0 tx=0x7fcf58010c90 comp rx=0 tx=0).stop 2024-09-13T16:47:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.174+0000 7fcf62000640 1 -- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c06ce80 msgr2=0x7fcf6c0c3a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:47:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.174+0000 7fcf62000640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c06ce80 0x7fcf6c0c3a10 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c1ea5f0 tx=0x7fcf58010c90 comp rx=0 tx=0).stop 2024-09-13T16:47:07.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.178+0000 7fcf62000640 1 -- 172.21.15.28:0/2549864584 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcf5c0113f0 con 0x7fcf6c06af90 2024-09-13T16:47:07.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.180+0000 7fcf62000640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fcf4c06a9d0 0x7fcf4c06ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.181+0000 7fcf62000640 1 -- 172.21.15.28:0/2549864584 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcf5c0894c0 con 0x7fcf6c06af90 2024-09-13T16:47:07.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.181+0000 7fcf69400640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fcf4c06a9d0 0x7fcf4c06ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.182+0000 7fcf69400640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fcf4c06a9d0 0x7fcf4c06ce90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcf5800e5b0 tx=0x7fcf58016000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.183+0000 7fcf6b200640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c1f3e00 0x7fcf6c1f41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.183+0000 7fcf6b200640 1 -- 172.21.15.28:0/2549864584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c15cb40 con 0x7fcf6c1f3e00 2024-09-13T16:47:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.184+0000 7fcf6a800640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c1f3e00 0x7fcf6c1f41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.184+0000 7fcf6a800640 1 -- 172.21.15.28:0/2549864584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcf6c06bd30 con 0x7fcf6c1f3e00 2024-09-13T16:47:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.185+0000 7fcf6a800640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c1f3e00 0x7fcf6c1f41e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcf64033000 tx=0x7fcf64004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.185+0000 7fcf62000640 1 -- 172.21.15.28:0/2549864584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6403b820 con 0x7fcf6c1f3e00 2024-09-13T16:47:07.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.186+0000 7fcf62000640 1 -- 172.21.15.28:0/2549864584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4459 (secure 0 0 0) 0x7fcf64009510 con 0x7fcf6c1f3e00 2024-09-13T16:47:07.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.186+0000 7fcf62000640 1 -- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c1f3e00 msgr2=0x7fcf6c1f41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.186+0000 7fcf62000640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c1f3e00 0x7fcf6c1f41e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcf64033000 tx=0x7fcf64004a40 comp rx=0 tx=0).stop 2024-09-13T16:47:07.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.186+0000 7fcf62000640 1 -- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c1f3e00 msgr2=0x7fcf6c1f41e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:47:07.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.186+0000 7fcf62000640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf6c1f3e00 0x7fcf6c1f41e0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcf64033000 tx=0x7fcf64004a40 comp rx=0 tx=0).stop 2024-09-13T16:47:07.188 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:47:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:47:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:47:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "leader", 2024-09-13T16:47:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 22, 2024-09-13T16:47:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:47:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:47:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:47:07.188 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:47:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 2, 2024-09-13T16:47:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:47:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:47:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:47:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:07.189 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:47:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:47:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:07.190 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:47:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:47:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:47:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:47:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:47:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:47:07.191 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:47:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:47:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:47:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:47:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:47:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:47:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:47:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:47:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:47:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:47:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:47:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:07.192 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:47:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:47:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.194 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:47:07.194 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:47:07.194 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:07.194 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:07.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.194 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:07.194 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:47:07.194 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.194 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:07.194 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:47:07.194 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.195 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.195 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.195 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:47:07.195 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:47:07.195 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:07.195 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:07.195 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:07.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.195 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:47:07.195 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:47:07.195 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:47:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:47:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:47:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:47:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:47:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:47:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:47:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:47:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:47:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:47:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:47:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:47:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:47:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:07.199 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:47:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:47:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.200 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:47:07.200 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:47:07.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.186+0000 7fcf6b200640 1 -- 172.21.15.28:0/2549864584 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fcf4c06a9d0 msgr2=0x7fcf4c06ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.186+0000 7fcf6b200640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fcf4c06a9d0 0x7fcf4c06ce90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcf5800e5b0 tx=0x7fcf58016000 comp rx=0 tx=0).stop 2024-09-13T16:47:07.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.186+0000 7fcf6b200640 1 -- 172.21.15.28:0/2549864584 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf6c06af90 msgr2=0x7fcf6c157420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.186+0000 7fcf6b200640 1 --2- 172.21.15.28:0/2549864584 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcf6c06af90 0x7fcf6c157420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00c820 tx=0x7fcf5c00cce0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.186+0000 7fcf6a800640 1 -- 172.21.15.28:0/2549864584 reap_dead start 2024-09-13T16:47:07.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.186+0000 7fcf6b200640 1 -- 172.21.15.28:0/2549864584 shutdown_connections 2024-09-13T16:47:07.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.186+0000 7fcf6b200640 1 -- 172.21.15.28:0/2549864584 >> 172.21.15.28:0/2549864584 conn(0x7fcf6c06a5a0 msgr2=0x7fcf6c1e4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:07.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.187+0000 7fcf6b200640 1 -- 172.21.15.28:0/2549864584 shutdown_connections 2024-09-13T16:47:07.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.187+0000 7fcf6b200640 1 -- 172.21.15.28:0/2549864584 wait complete. 2024-09-13T16:47:07.208 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:47:07.208 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-13T16:47:07.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T16:47:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.433+0000 7f43dc600640 1 Processor -- start 2024-09-13T16:47:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.433+0000 7f43dc600640 1 -- start start 2024-09-13T16:47:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.434+0000 7f43dc600640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f43d4152d70 0x7f43d40c7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.435+0000 7f43dc600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43d4071ce0 0x7f43d406ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.435+0000 7f43d9c00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f43d4152d70 0x7f43d40c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.435+0000 7f43dc600640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d415cf00 con 0x7f43d406a8b0 2024-09-13T16:47:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.435+0000 7f43dc600640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d40c59f0 con 0x7f43d4152d70 2024-09-13T16:47:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.435+0000 7f43dc600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d4153150 con 0x7f43d4071ce0 2024-09-13T16:47:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.435+0000 7f43d9c00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f43d4152d70 0x7f43d40c7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:47874/0 (socket says 172.21.15.28:47874) 2024-09-13T16:47:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.435+0000 7f43d9c00640 1 -- 172.21.15.28:0/2951120071 learned_addr learned my addr 172.21.15.28:0/2951120071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.435+0000 7f43da600640 1 --2- 172.21.15.28:0/2951120071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43d4071ce0 0x7f43d406ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.436+0000 7f43d9c00640 1 -- 172.21.15.28:0/2951120071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43d4071ce0 msgr2=0x7f43d406ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:47:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.436+0000 7f43d9c00640 1 --2- 172.21.15.28:0/2951120071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43d4071ce0 0x7f43d406ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.436+0000 7f43d9c00640 1 -- 172.21.15.28:0/2951120071 >> v1:172.21.15.191:6789/0 conn(0x7f43d406a8b0 legacy=0x7f43d4069f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:47:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.436+0000 7f43d9c00640 1 -- 172.21.15.28:0/2951120071 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d406a620 con 0x7f43d4152d70 2024-09-13T16:47:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.437+0000 7f43d9c00640 1 --2- 172.21.15.28:0/2951120071 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d4152d70 0x7f43d40c7070 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f43c8009870 tx=0x7f43c802a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca413c3458ada4a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.438+0000 7f43d3e00640 1 -- 172.21.15.28:0/2951120071 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c802ae10 con 0x7f43d4152d70 2024-09-13T16:47:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.438+0000 7f43d3e00640 1 -- 172.21.15.28:0/2951120071 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c8032440 con 0x7f43d4152d70 2024-09-13T16:47:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.438+0000 7f43d3e00640 1 -- 172.21.15.28:0/2951120071 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c80327d0 con 0x7f43d4152d70 2024-09-13T16:47:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.439+0000 7f43dc600640 1 -- 172.21.15.28:0/2951120071 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d4152d70 msgr2=0x7f43d40c7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.439+0000 7f43dc600640 1 --2- 172.21.15.28:0/2951120071 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d4152d70 0x7f43d40c7070 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f43c8009870 tx=0x7f43c802a100 comp rx=0 tx=0).stop 2024-09-13T16:47:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.439+0000 7f43dc600640 1 -- 172.21.15.28:0/2951120071 shutdown_connections 2024-09-13T16:47:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.439+0000 7f43dc600640 1 --2- 172.21.15.28:0/2951120071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43d4071ce0 0x7f43d406ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.439+0000 7f43dc600640 1 --2- 172.21.15.28:0/2951120071 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d4152d70 0x7f43d40c7070 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.439+0000 7f43dc600640 1 -- 172.21.15.28:0/2951120071 >> 172.21.15.28:0/2951120071 conn(0x7f43d40cb890 msgr2=0x7f43d40cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.439+0000 7f43dc600640 1 -- 172.21.15.28:0/2951120071 shutdown_connections 2024-09-13T16:47:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.439+0000 7f43dc600640 1 -- 172.21.15.28:0/2951120071 wait complete. 2024-09-13T16:47:07.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.440+0000 7f43dc600640 1 Processor -- start 2024-09-13T16:47:07.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.441+0000 7f43dc600640 1 -- start start 2024-09-13T16:47:07.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.441+0000 7f43dc600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43d406a8b0 0x7f43d414aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.441+0000 7f43dc600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43d4071ce0 0x7f43d41497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.441+0000 7f43dc600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d4152d70 0x7f43d41505e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.441+0000 7f43dc600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d4149d20 con 0x7f43d406a8b0 2024-09-13T16:47:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.441+0000 7f43dc600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d41ee730 con 0x7f43d4152d70 2024-09-13T16:47:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.441+0000 7f43dc600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d414edb0 con 0x7f43d4071ce0 2024-09-13T16:47:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.442+0000 7f43d9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43d406a8b0 0x7f43d414aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.442+0000 7f43d9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43d406a8b0 0x7f43d414aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53328/0 (socket says 172.21.15.28:53328) 2024-09-13T16:47:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.442+0000 7f43d9c00640 1 -- 172.21.15.28:0/84897264 learned_addr learned my addr 172.21.15.28:0/84897264 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.442+0000 7f43da600640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d4152d70 0x7f43d41505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.442+0000 7f43d9200640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43d4071ce0 0x7f43d41497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.442+0000 7f43d9200640 1 -- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d4152d70 msgr2=0x7f43d41505e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.442+0000 7f43d9200640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d4152d70 0x7f43d41505e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.442+0000 7f43d9200640 1 -- 172.21.15.28:0/84897264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43d406a8b0 msgr2=0x7f43d414aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.442+0000 7f43d9200640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43d406a8b0 0x7f43d414aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.442+0000 7f43d9200640 1 -- 172.21.15.28:0/84897264 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c80094d0 con 0x7f43d4071ce0 2024-09-13T16:47:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.442+0000 7f43da600640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d4152d70 0x7f43d41505e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:47:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.443+0000 7f43d9200640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43d4071ce0 0x7f43d41497e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f43c000b3e0 tx=0x7f43c000b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.443+0000 7f43d1600640 1 -- 172.21.15.28:0/84897264 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c0004020 con 0x7f43d4071ce0 2024-09-13T16:47:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.443+0000 7f43d1600640 1 -- 172.21.15.28:0/84897264 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c0004180 con 0x7f43d4071ce0 2024-09-13T16:47:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.443+0000 7f43dc600640 1 -- 172.21.15.28:0/84897264 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d414b8c0 con 0x7f43d4071ce0 2024-09-13T16:47:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.443+0000 7f43d1600640 1 -- 172.21.15.28:0/84897264 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c0011630 con 0x7f43d4071ce0 2024-09-13T16:47:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.443+0000 7f43dc600640 1 -- 172.21.15.28:0/84897264 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d414b300 con 0x7f43d4071ce0 2024-09-13T16:47:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.444+0000 7f43dc600640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d414e620 0x7f43d41eeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.444+0000 7f43dc600640 1 -- 172.21.15.28:0/84897264 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d4073c20 con 0x7f43d414e620 2024-09-13T16:47:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.445+0000 7f43d9c00640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d414e620 0x7f43d41eeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.445+0000 7f43d9c00640 1 -- 172.21.15.28:0/84897264 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f43d40c5470 con 0x7f43d414e620 2024-09-13T16:47:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.446+0000 7f43d9c00640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d414e620 0x7f43d41eeba0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f43c8035040 tx=0x7f43c80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:07.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.446+0000 7f43d1600640 1 -- 172.21.15.28:0/84897264 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f43c0011790 con 0x7f43d4071ce0 2024-09-13T16:47:07.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.447+0000 7f43d1600640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f43b406ac30 0x7f43b406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.447+0000 7f43d1600640 1 -- 172.21.15.28:0/84897264 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f43c0088f40 con 0x7f43d4071ce0 2024-09-13T16:47:07.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.448+0000 7f43da600640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f43b406ac30 0x7f43b406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.448+0000 7f43d1600640 1 -- 172.21.15.28:0/84897264 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c803c910 con 0x7f43d414e620 2024-09-13T16:47:07.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.448+0000 7f43d1600640 1 -- 172.21.15.28:0/84897264 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f43c803ca70 con 0x7f43d414e620 2024-09-13T16:47:07.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.448+0000 7f43d1600640 1 -- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d414e620 msgr2=0x7f43d41eeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.448+0000 7f43d1600640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d414e620 0x7f43d41eeba0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f43c8035040 tx=0x7f43c80046c0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.448+0000 7f43d1600640 1 -- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d414e620 msgr2=0x7f43d41eeba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:47:07.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.448+0000 7f43d1600640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d414e620 0x7f43d41eeba0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.449+0000 7f43da600640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f43b406ac30 0x7f43b406d0f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f43cc009fd0 tx=0x7f43cc00b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:07.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.455+0000 7f43dc600640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d4150b20 0x7f43d41ef0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.455+0000 7f43dc600640 1 -- 172.21.15.28:0/84897264 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d4072290 con 0x7f43d4150b20 2024-09-13T16:47:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.455+0000 7f43d9c00640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d4150b20 0x7f43d41ef0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.455+0000 7f43d9c00640 1 -- 172.21.15.28:0/84897264 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f43d406b0b0 con 0x7f43d4150b20 2024-09-13T16:47:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.456+0000 7f43d9c00640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d4150b20 0x7f43d41ef0e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f43c802a140 tx=0x7f43c8008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:07.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.456+0000 7f43d1600640 1 -- 172.21.15.28:0/84897264 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c804c2b0 con 0x7f43d4150b20 2024-09-13T16:47:07.458 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:47:07.458 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:47:07.458 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:47:07.458 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:47:07.458 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 22, 2024-09-13T16:47:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:47:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:47:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:47:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:47:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 2, 2024-09-13T16:47:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:47:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:47:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:47:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:07.459 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:47:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:47:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:07.460 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:47:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:47:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:47:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:47:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:47:07.461 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:47:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:47:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:47:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:47:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:47:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:47:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:47:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:47:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:47:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:47:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:47:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:47:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:07.462 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:47:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.463 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:47:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:47:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:47:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:47:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:07.465 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:47:07.465 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.465 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.465 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.465 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:47:07.465 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:47:07.465 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:07.465 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:07.465 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:47:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:47:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:47:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:07.466 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:47:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:47:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:47:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:47:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:47:07.467 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:47:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:47:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.469 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:47:07.469 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:47:07.469 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:07.469 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:07.469 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:07.469 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.469 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.469 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:47:07.469 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:47:07.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:47:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:47:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:47:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:47:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:47:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:47:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:47:07.471 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.472 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.472 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:47:07.472 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:47:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.456+0000 7f43d1600640 1 -- 172.21.15.28:0/84897264 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4789 (secure 0 0 0) 0x7f43c8003da0 con 0x7f43d4150b20 2024-09-13T16:47:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.456+0000 7f43d1600640 1 -- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d4150b20 msgr2=0x7f43d41ef0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.456+0000 7f43d1600640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d4150b20 0x7f43d41ef0e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f43c802a140 tx=0x7f43c8008030 comp rx=0 tx=0).stop 2024-09-13T16:47:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.456+0000 7f43d1600640 1 -- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d4150b20 msgr2=0x7f43d41ef0e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:47:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.456+0000 7f43d1600640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43d4150b20 0x7f43d41ef0e0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f43c802a140 tx=0x7f43c8008030 comp rx=0 tx=0).stop 2024-09-13T16:47:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.457+0000 7f43dc600640 1 -- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f43b406ac30 msgr2=0x7f43b406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.457+0000 7f43dc600640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f43b406ac30 0x7f43b406d0f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f43cc009fd0 tx=0x7f43cc00b040 comp rx=0 tx=0).stop 2024-09-13T16:47:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.457+0000 7f43dc600640 1 -- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43d4071ce0 msgr2=0x7f43d41497e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.457+0000 7f43dc600640 1 --2- 172.21.15.28:0/84897264 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43d4071ce0 0x7f43d41497e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f43c000b3e0 tx=0x7f43c000b8a0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.457+0000 7f43da600640 1 -- 172.21.15.28:0/84897264 reap_dead start 2024-09-13T16:47:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.457+0000 7f43da600640 1 -- 172.21.15.28:0/84897264 reap_dead start 2024-09-13T16:47:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.457+0000 7f43dc600640 1 -- 172.21.15.28:0/84897264 shutdown_connections 2024-09-13T16:47:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.457+0000 7f43dc600640 1 -- 172.21.15.28:0/84897264 >> 172.21.15.28:0/84897264 conn(0x7f43d40cb890 msgr2=0x7f43d415bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.458+0000 7f43dc600640 1 -- 172.21.15.28:0/84897264 shutdown_connections 2024-09-13T16:47:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.459+0000 7f43dc600640 1 -- 172.21.15.28:0/84897264 wait complete. 2024-09-13T16:47:07.482 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:47:07.482 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-13T16:47:07.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T16:47:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.739+0000 7fe266000640 1 Processor -- start 2024-09-13T16:47:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.739+0000 7fe266000640 1 -- start start 2024-09-13T16:47:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.741+0000 7fe266000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe26014ebc0 0x7fe260155390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.741+0000 7fe266000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe26006cf10 0x7fe26006d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.741+0000 7fe266000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe26006b970 0x7fe26006be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.741+0000 7fe266000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe26006e1f0 con 0x7fe26006b970 2024-09-13T16:47:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.741+0000 7fe266000640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2601558d0 con 0x7fe26006cf10 2024-09-13T16:47:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.741+0000 7fe266000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2600c6a70 con 0x7fe26014ebc0 2024-09-13T16:47:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.742+0000 7fe25fe00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe26006cf10 0x7fe26006d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.742+0000 7fe25fe00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe26006cf10 0x7fe26006d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:47890/0 (socket says 172.21.15.28:47890) 2024-09-13T16:47:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.742+0000 7fe25fe00640 1 -- 172.21.15.28:0/3561827906 learned_addr learned my addr 172.21.15.28:0/3561827906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.743+0000 7fe265600640 1 --2- 172.21.15.28:0/3561827906 >> v2:172.21.15.28:3300/0 conn(0x7fe26006b970 0x7fe26006be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.743+0000 7fe265600640 1 -- 172.21.15.28:0/3561827906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe26014ebc0 msgr2=0x7fe260155390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-13T16:47:07.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.743+0000 7fe265600640 1 --2- 172.21.15.28:0/3561827906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe26014ebc0 0x7fe260155390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.743+0000 7fe265600640 1 -- 172.21.15.28:0/3561827906 >> v2:172.21.15.191:3300/0 conn(0x7fe26006cf10 msgr2=0x7fe26006d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:47:07.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.743+0000 7fe265600640 1 --2- 172.21.15.28:0/3561827906 >> v2:172.21.15.191:3300/0 conn(0x7fe26006cf10 0x7fe26006d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.743+0000 7fe265600640 1 -- 172.21.15.28:0/3561827906 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2600c5390 con 0x7fe26006b970 2024-09-13T16:47:07.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.745+0000 7fe265600640 1 --2- 172.21.15.28:0/3561827906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe26006b970 0x7fe26006be00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe2600c7eb0 tx=0x7fe25802eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc4c85a5d6860672 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:07.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.748+0000 7fe25f400640 1 -- 172.21.15.28:0/3561827906 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25800d4e0 con 0x7fe26006b970 2024-09-13T16:47:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.748+0000 7fe25f400640 1 -- 172.21.15.28:0/3561827906 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25800b740 con 0x7fe26006b970 2024-09-13T16:47:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.748+0000 7fe25f400640 1 -- 172.21.15.28:0/3561827906 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25800bb10 con 0x7fe26006b970 2024-09-13T16:47:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.749+0000 7fe266000640 1 -- 172.21.15.28:0/3561827906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe26006b970 msgr2=0x7fe26006be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.749+0000 7fe266000640 1 --2- 172.21.15.28:0/3561827906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe26006b970 0x7fe26006be00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe2600c7eb0 tx=0x7fe25802eaf0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.752+0000 7fe266000640 1 -- 172.21.15.28:0/3561827906 shutdown_connections 2024-09-13T16:47:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.752+0000 7fe266000640 1 --2- 172.21.15.28:0/3561827906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe26006b970 0x7fe26006be00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.752+0000 7fe266000640 1 --2- 172.21.15.28:0/3561827906 >> v2:172.21.15.191:3300/0 conn(0x7fe26006cf10 0x7fe26006d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.752+0000 7fe266000640 1 --2- 172.21.15.28:0/3561827906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe26014ebc0 0x7fe260155390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.752+0000 7fe266000640 1 -- 172.21.15.28:0/3561827906 >> 172.21.15.28:0/3561827906 conn(0x7fe2600ca2f0 msgr2=0x7fe2600ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.752+0000 7fe266000640 1 -- 172.21.15.28:0/3561827906 shutdown_connections 2024-09-13T16:47:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.752+0000 7fe266000640 1 -- 172.21.15.28:0/3561827906 wait complete. 2024-09-13T16:47:07.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.760+0000 7fe266000640 1 Processor -- start 2024-09-13T16:47:07.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.760+0000 7fe266000640 1 -- start start 2024-09-13T16:47:07.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.762+0000 7fe266000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe26006b970 0x7fe260152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.762+0000 7fe266000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe26006cf10 0x7fe260153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.762+0000 7fe266000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe26014ebc0 0x7fe260149160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.762+0000 7fe266000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe260151c10 con 0x7fe26006cf10 2024-09-13T16:47:07.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.762+0000 7fe266000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2601f5980 con 0x7fe26006b970 2024-09-13T16:47:07.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.762+0000 7fe266000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2601f4b40 con 0x7fe26014ebc0 2024-09-13T16:47:07.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.764+0000 7fe25fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe26006cf10 0x7fe260153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.764+0000 7fe25fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe26006cf10 0x7fe260153010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53350/0 (socket says 172.21.15.28:53350) 2024-09-13T16:47:07.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.764+0000 7fe25fe00640 1 -- 172.21.15.28:0/3208274512 learned_addr learned my addr 172.21.15.28:0/3208274512 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:07.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.764+0000 7fe25fe00640 1 -- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe26014ebc0 msgr2=0x7fe260149160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:47:07.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.764+0000 7fe25fe00640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe26014ebc0 0x7fe260149160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.764+0000 7fe25fe00640 1 -- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe26006b970 msgr2=0x7fe260152ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:47:07.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.764+0000 7fe25fe00640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe26006b970 0x7fe260152ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.764+0000 7fe25fe00640 1 -- 172.21.15.28:0/3208274512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25800f050 con 0x7fe26006cf10 2024-09-13T16:47:07.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.764+0000 7fe25fe00640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe26006cf10 0x7fe260153010 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00b6b0 tx=0x7fe24c00bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:07.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.765+0000 7fe25cc00640 1 -- 172.21.15.28:0/3208274512 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c01a070 con 0x7fe26006cf10 2024-09-13T16:47:07.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.765+0000 7fe266000640 1 -- 172.21.15.28:0/3208274512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2601f4e00 con 0x7fe26006cf10 2024-09-13T16:47:07.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.765+0000 7fe266000640 1 -- 172.21.15.28:0/3208274512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2601f53c0 con 0x7fe26006cf10 2024-09-13T16:47:07.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.765+0000 7fe25cc00640 1 -- 172.21.15.28:0/3208274512 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24c00eaf0 con 0x7fe26006cf10 2024-09-13T16:47:07.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.765+0000 7fe25cc00640 1 -- 172.21.15.28:0/3208274512 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c00ee60 con 0x7fe26006cf10 2024-09-13T16:47:07.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.766+0000 7fe266000640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe23c002a30 0x7fe23c004ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.766+0000 7fe266000640 1 -- 172.21.15.28:0/3208274512 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe23c007e80 con 0x7fe23c002a30 2024-09-13T16:47:07.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.767+0000 7fe264c00640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe23c002a30 0x7fe23c004ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.768+0000 7fe264c00640 1 -- 172.21.15.28:0/3208274512 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe23c008040 con 0x7fe23c002a30 2024-09-13T16:47:07.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.768+0000 7fe264c00640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe23c002a30 0x7fe23c004ef0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe25400b340 tx=0x7fe25400b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:07.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.768+0000 7fe25cc00640 1 -- 172.21.15.28:0/3208274512 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe254011830 con 0x7fe23c002a30 2024-09-13T16:47:07.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.769+0000 7fe25cc00640 1 -- 172.21.15.28:0/3208274512 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe24c016470 con 0x7fe26006cf10 2024-09-13T16:47:07.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.770+0000 7fe25cc00640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe24007bcb0 0x7fe24007e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.770+0000 7fe25cc00640 1 -- 172.21.15.28:0/3208274512 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe24c010040 con 0x7fe26006cf10 2024-09-13T16:47:07.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.770+0000 7fe25cc00640 1 -- 172.21.15.28:0/3208274512 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fe254011990 con 0x7fe23c002a30 2024-09-13T16:47:07.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.770+0000 7fe25cc00640 1 -- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe23c002a30 msgr2=0x7fe23c004ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.770+0000 7fe25cc00640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe23c002a30 0x7fe23c004ef0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe25400b340 tx=0x7fe25400b800 comp rx=0 tx=0).stop 2024-09-13T16:47:07.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.770+0000 7fe265600640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe24007bcb0 0x7fe24007e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.771+0000 7fe25cc00640 1 -- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe23c002a30 msgr2=0x7fe23c004ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:47:07.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.771+0000 7fe25cc00640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe23c002a30 0x7fe23c004ef0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:07.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.773+0000 7fe265600640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe24007bcb0 0x7fe24007e170 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe258038000 tx=0x7fe25800d000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.780+0000 7fe266000640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe23c006ef0 0x7fe23c007380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.780+0000 7fe266000640 1 -- 172.21.15.28:0/3208274512 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe23c00a3e0 con 0x7fe23c006ef0 2024-09-13T16:47:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.781+0000 7fe264c00640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe23c006ef0 0x7fe23c007380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.781+0000 7fe264c00640 1 -- 172.21.15.28:0/3208274512 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe23c0078c0 con 0x7fe23c006ef0 2024-09-13T16:47:07.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.782+0000 7fe264c00640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe23c006ef0 0x7fe23c007380 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe25400b340 tx=0x7fe254004640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:07.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.782+0000 7fe25cc00640 1 -- 172.21.15.28:0/3208274512 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25401c310 con 0x7fe23c006ef0 2024-09-13T16:47:07.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.782+0000 7fe25cc00640 1 -- 172.21.15.28:0/3208274512 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+5098 (secure 0 0 0) 0x7fe2540263e0 con 0x7fe23c006ef0 2024-09-13T16:47:07.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.782+0000 7fe25cc00640 1 -- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe23c006ef0 msgr2=0x7fe23c007380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.782+0000 7fe25cc00640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe23c006ef0 0x7fe23c007380 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe25400b340 tx=0x7fe254004640 comp rx=0 tx=0).stop 2024-09-13T16:47:07.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.782+0000 7fe25cc00640 1 -- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe23c006ef0 msgr2=0x7fe23c007380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:47:07.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.782+0000 7fe25cc00640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe23c006ef0 0x7fe23c007380 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe25400b340 tx=0x7fe254004640 comp rx=0 tx=0).stop 2024-09-13T16:47:07.784 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:47:07.784 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:47:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:47:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:47:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 22, 2024-09-13T16:47:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:47:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:47:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:47:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:47:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 2, 2024-09-13T16:47:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:47:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:47:07.785 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:47:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:47:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:47:07.786 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:47:07.787 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:47:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:47:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:47:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:47:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:47:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:47:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:47:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:47:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:47:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:47:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:47:07.788 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:47:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:47:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:47:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:47:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:47:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:07.789 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:47:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:47:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:47:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:47:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.790 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:47:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:47:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:47:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:47:07.791 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:47:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:47:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:47:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:47:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:47:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:47:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:07.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:47:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:47:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:47:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:07.794 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:47:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:47:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:47:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:07.795 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:47:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:47:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:47:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:47:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.796 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 5 2024-09-13T16:47:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:47:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 6 2024-09-13T16:47:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:47:07.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x2f018fb87aa4aafe", 2024-09-13T16:47:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "luminous", 2024-09-13T16:47:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:47:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4 2024-09-13T16:47:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": [ 2024-09-13T16:47:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:47:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:07.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:07.800 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:47:07.800 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:47:07.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.783+0000 7fe266000640 1 -- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe24007bcb0 msgr2=0x7fe24007e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.783+0000 7fe266000640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe24007bcb0 0x7fe24007e170 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe258038000 tx=0x7fe25800d000 comp rx=0 tx=0).stop 2024-09-13T16:47:07.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.783+0000 7fe266000640 1 -- 172.21.15.28:0/3208274512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe26006cf10 msgr2=0x7fe260153010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:07.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.783+0000 7fe266000640 1 --2- 172.21.15.28:0/3208274512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe26006cf10 0x7fe260153010 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00b6b0 tx=0x7fe24c00bb70 comp rx=0 tx=0).stop 2024-09-13T16:47:07.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.783+0000 7fe265600640 1 -- 172.21.15.28:0/3208274512 reap_dead start 2024-09-13T16:47:07.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.783+0000 7fe265600640 1 -- 172.21.15.28:0/3208274512 reap_dead start 2024-09-13T16:47:07.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.783+0000 7fe266000640 1 -- 172.21.15.28:0/3208274512 shutdown_connections 2024-09-13T16:47:07.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.783+0000 7fe266000640 1 -- 172.21.15.28:0/3208274512 >> 172.21.15.28:0/3208274512 conn(0x7fe2600ca2f0 msgr2=0x7fe2600c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:07.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.783+0000 7fe266000640 1 -- 172.21.15.28:0/3208274512 shutdown_connections 2024-09-13T16:47:07.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:07.783+0000 7fe266000640 1 -- 172.21.15.28:0/3208274512 wait complete. 2024-09-13T16:47:07.802 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:47:07.802 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-13T16:47:07.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-13T16:47:08.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.037+0000 7f31f3000640 1 Processor -- start 2024-09-13T16:47:08.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.037+0000 7f31f3000640 1 -- start start 2024-09-13T16:47:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.038+0000 7f31f3000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f31ec152d10 0x7f31ec0cb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.038+0000 7f31f3000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ec0cadc0 con 0x7f31ec0cb790 2024-09-13T16:47:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.038+0000 7f31f3000640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ec06bc40 con 0x7f31ec15cc50 2024-09-13T16:47:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.038+0000 7f31f3000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31ec0ca4d0 con 0x7f31ec152d10 2024-09-13T16:47:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.038+0000 7f31eb400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f31ec0cb790 0x7f31ec15c0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43400/0 (socket says 172.21.15.28:43400) 2024-09-13T16:47:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.038+0000 7f31eb400640 1 -- 172.21.15.28:0/4214056610 learned_addr learned my addr 172.21.15.28:0/4214056610 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:47:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.038+0000 7f31ebe00640 1 --2- 172.21.15.28:0/4214056610 >> v2:172.21.15.28:3300/0 conn(0x7f31ec152d10 0x7f31ec0cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.039+0000 7f31eaa00640 1 -- 172.21.15.28:0/4214056610 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637109031 0 0) 0x7f31ec06bc40 con 0x7f31ec15cc50 2024-09-13T16:47:08.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.039+0000 7f31eaa00640 1 -- 172.21.15.28:0/4214056610 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31d0003520 con 0x7f31ec15cc50 2024-09-13T16:47:08.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.039+0000 7f31eaa00640 1 -- 172.21.15.28:0/4214056610 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652081687 0 0) 0x7f31ec0cadc0 con 0x7f31ec0cb790 2024-09-13T16:47:08.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.039+0000 7f31eaa00640 1 -- 172.21.15.28:0/4214056610 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31ec06bc40 con 0x7f31ec0cb790 2024-09-13T16:47:08.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.039+0000 7f31ebe00640 1 -- 172.21.15.28:0/4214056610 >> v1:172.21.15.191:6789/0 conn(0x7f31ec15cc50 legacy=0x7f31ec06acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:08.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.039+0000 7f31ebe00640 1 -- 172.21.15.28:0/4214056610 >> v1:172.21.15.28:6789/0 conn(0x7f31ec0cb790 legacy=0x7f31ec15c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:08.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.039+0000 7f31ebe00640 1 -- 172.21.15.28:0/4214056610 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec06b960 con 0x7f31ec152d10 2024-09-13T16:47:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.040+0000 7f31ebe00640 1 --2- 172.21.15.28:0/4214056610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ec152d10 0x7f31ec0cb250 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f31dc009800 tx=0x7f31dc02a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87e9ca90ec3f17df server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.040+0000 7f31eaa00640 1 -- 172.21.15.28:0/4214056610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31dc032440 con 0x7f31ec152d10 2024-09-13T16:47:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.040+0000 7f31eaa00640 1 -- 172.21.15.28:0/4214056610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31dc0325a0 con 0x7f31ec152d10 2024-09-13T16:47:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.041+0000 7f31eaa00640 1 -- 172.21.15.28:0/4214056610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31dc032910 con 0x7f31ec152d10 2024-09-13T16:47:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.041+0000 7f31f3000640 1 -- 172.21.15.28:0/4214056610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ec152d10 msgr2=0x7f31ec0cb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.041+0000 7f31f3000640 1 --2- 172.21.15.28:0/4214056610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ec152d10 0x7f31ec0cb250 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f31dc009800 tx=0x7f31dc02a200 comp rx=0 tx=0).stop 2024-09-13T16:47:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.041+0000 7f31f3000640 1 -- 172.21.15.28:0/4214056610 shutdown_connections 2024-09-13T16:47:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.041+0000 7f31f3000640 1 --2- 172.21.15.28:0/4214056610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ec152d10 0x7f31ec0cb250 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.041+0000 7f31f3000640 1 -- 172.21.15.28:0/4214056610 >> 172.21.15.28:0/4214056610 conn(0x7f31ec065c50 msgr2=0x7f31ec066060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.041+0000 7f31f3000640 1 -- 172.21.15.28:0/4214056610 shutdown_connections 2024-09-13T16:47:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.041+0000 7f31f3000640 1 -- 172.21.15.28:0/4214056610 wait complete. 2024-09-13T16:47:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.043+0000 7f31f3000640 1 Processor -- start 2024-09-13T16:47:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.043+0000 7f31f3000640 1 -- start start 2024-09-13T16:47:08.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.044+0000 7f31f3000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ec0cb790 0x7f31ec1e1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:08.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.044+0000 7f31f3000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31ec152d10 0x7f31ec14ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:08.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.044+0000 7f31f3000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31ec15cc50 0x7f31ec0ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:08.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.044+0000 7f31f3000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ec14c230 con 0x7f31ec0cb790 2024-09-13T16:47:08.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.044+0000 7f31f3000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ec0c85f0 con 0x7f31ec152d10 2024-09-13T16:47:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.044+0000 7f31f3000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31ec14c460 con 0x7f31ec15cc50 2024-09-13T16:47:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.044+0000 7f31ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ec0cb790 0x7f31ec1e1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.044+0000 7f31ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ec0cb790 0x7f31ec1e1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53356/0 (socket says 172.21.15.28:53356) 2024-09-13T16:47:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.044+0000 7f31eb400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31ec152d10 0x7f31ec14ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.044+0000 7f31ebe00640 1 -- 172.21.15.28:0/1126866692 learned_addr learned my addr 172.21.15.28:0/1126866692 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.044+0000 7f31f1000640 1 --2- 172.21.15.28:0/1126866692 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31ec15cc50 0x7f31ec0ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.045+0000 7f31ebe00640 1 -- 172.21.15.28:0/1126866692 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31ec15cc50 msgr2=0x7f31ec0ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.045+0000 7f31ebe00640 1 --2- 172.21.15.28:0/1126866692 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31ec15cc50 0x7f31ec0ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.045+0000 7f31ebe00640 1 -- 172.21.15.28:0/1126866692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31ec152d10 msgr2=0x7f31ec14ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.045+0000 7f31ebe00640 1 --2- 172.21.15.28:0/1126866692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31ec152d10 0x7f31ec14ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.045+0000 7f31ebe00640 1 -- 172.21.15.28:0/1126866692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31dc009400 con 0x7f31ec0cb790 2024-09-13T16:47:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.045+0000 7f31ebe00640 1 --2- 172.21.15.28:0/1126866692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ec0cb790 0x7f31ec1e1d40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f31dc009920 tx=0x7f31dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.045+0000 7f31e3e00640 1 -- 172.21.15.28:0/1126866692 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31dc006e10 con 0x7f31ec0cb790 2024-09-13T16:47:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.045+0000 7f31e3e00640 1 -- 172.21.15.28:0/1126866692 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31dc0044e0 con 0x7f31ec0cb790 2024-09-13T16:47:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.045+0000 7f31f3000640 1 -- 172.21.15.28:0/1126866692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ec1e2320 con 0x7f31ec0cb790 2024-09-13T16:47:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.045+0000 7f31e3e00640 1 -- 172.21.15.28:0/1126866692 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31dc003610 con 0x7f31ec0cb790 2024-09-13T16:47:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.046+0000 7f31f3000640 1 -- 172.21.15.28:0/1126866692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31ec14bf90 con 0x7f31ec0cb790 2024-09-13T16:47:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.047+0000 7f31e1600640 1 -- 172.21.15.28:0/1126866692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31ec1f28e0 con 0x7f31ec0cb790 2024-09-13T16:47:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.048+0000 7f31e3e00640 1 -- 172.21.15.28:0/1126866692 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f31dc003770 con 0x7f31ec0cb790 2024-09-13T16:47:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.048+0000 7f31e3e00640 1 --2- 172.21.15.28:0/1126866692 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f31cc06ac00 0x7f31cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.049+0000 7f31e3e00640 1 -- 172.21.15.28:0/1126866692 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f31dc0a9df0 con 0x7f31ec0cb790 2024-09-13T16:47:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.049+0000 7f31eb400640 1 --2- 172.21.15.28:0/1126866692 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f31cc06ac00 0x7f31cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.049+0000 7f31eb400640 1 --2- 172.21.15.28:0/1126866692 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f31cc06ac00 0x7f31cc06d0c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0c8a10 tx=0x7f31d8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:08.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.054+0000 7f31e3e00640 1 -- 172.21.15.28:0/1126866692 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f31ec1f28e0 con 0x7f31ec0cb790 2024-09-13T16:47:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.394+0000 7f31e1600640 1 -- 172.21.15.28:0/1126866692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f31ec1f28e0 con 0x7f31ec0cb790 2024-09-13T16:47:08.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.397+0000 7f31e3e00640 1 -- 172.21.15.28:0/1126866692 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f31ec1f28e0 con 0x7f31ec0cb790 2024-09-13T16:47:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.400+0000 7f31e1600640 1 -- 172.21.15.28:0/1126866692 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f31cc06ac00 msgr2=0x7f31cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.400+0000 7f31e1600640 1 --2- 172.21.15.28:0/1126866692 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f31cc06ac00 0x7f31cc06d0c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0c8a10 tx=0x7f31d8009040 comp rx=0 tx=0).stop 2024-09-13T16:47:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.400+0000 7f31e1600640 1 -- 172.21.15.28:0/1126866692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ec0cb790 msgr2=0x7f31ec1e1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.400+0000 7f31e1600640 1 --2- 172.21.15.28:0/1126866692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ec0cb790 0x7f31ec1e1d40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f31dc009920 tx=0x7f31dc0046c0 comp rx=0 tx=0).stop 2024-09-13T16:47:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.401+0000 7f31e1600640 1 -- 172.21.15.28:0/1126866692 shutdown_connections 2024-09-13T16:47:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.401+0000 7f31e1600640 1 --2- 172.21.15.28:0/1126866692 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f31cc06ac00 0x7f31cc06d0c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.401+0000 7f31e1600640 1 --2- 172.21.15.28:0/1126866692 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f31ec15cc50 0x7f31ec0ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.401+0000 7f31e1600640 1 --2- 172.21.15.28:0/1126866692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f31ec152d10 0x7f31ec14ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.401+0000 7f31e1600640 1 --2- 172.21.15.28:0/1126866692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ec0cb790 0x7f31ec1e1d40 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.401+0000 7f31e1600640 1 -- 172.21.15.28:0/1126866692 >> 172.21.15.28:0/1126866692 conn(0x7f31ec065c50 msgr2=0x7f31ec14cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.404+0000 7f31e1600640 1 -- 172.21.15.28:0/1126866692 shutdown_connections 2024-09-13T16:47:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:08.407+0000 7f31e1600640 1 -- 172.21.15.28:0/1126866692 wait complete. 2024-09-13T16:47:08.434 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-09-13T16:47:09.416 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:47:09.416 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 initial writers = 311626.03 kB/sec 2024-09-13T16:47:09.416 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 initial writers = 311618.78 kB/sec 2024-09-13T16:47:09.416 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 311626.03 kB/sec 2024-09-13T16:47:09.416 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 311626.03 kB/sec 2024-09-13T16:47:09.416 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 311626.03 kB/sec 2024-09-13T16:47:09.416 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 10485760.00 kB 2024-09-13T16:47:18.434 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T16:47:18.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:47:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.697+0000 7f4953600640 1 Processor -- start 2024-09-13T16:47:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.697+0000 7f4953600640 1 -- start start 2024-09-13T16:47:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.698+0000 7f4953600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f494c1570a0 0x7f494c070040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.698+0000 7f4953600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494c15df30 con 0x7f494c157480 2024-09-13T16:47:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.699+0000 7f4953600640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494c161340 con 0x7f494c0c6270 2024-09-13T16:47:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.699+0000 7f4953600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f494c15d320 con 0x7f494c1570a0 2024-09-13T16:47:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.699+0000 7f494be00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f494c0c6270 0x7f494c0c6ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:48514/0 (socket says 172.21.15.28:48514) 2024-09-13T16:47:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.699+0000 7f494be00640 1 -- 172.21.15.28:0/1139238778 learned_addr learned my addr 172.21.15.28:0/1139238778 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:47:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.699+0000 7f494b400640 1 -- 172.21.15.28:0/1139238778 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115394879 0 0) 0x7f494c161340 con 0x7f494c0c6270 2024-09-13T16:47:18.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.699+0000 7f494b400640 1 -- 172.21.15.28:0/1139238778 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4934003520 con 0x7f494c0c6270 2024-09-13T16:47:18.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.699+0000 7f4950c00640 1 --2- 172.21.15.28:0/1139238778 >> v2:172.21.15.28:3300/0 conn(0x7f494c1570a0 0x7f494c070040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:18.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.700+0000 7f494b400640 1 -- 172.21.15.28:0/1139238778 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2946656237 0 0) 0x7f4934003520 con 0x7f494c0c6270 2024-09-13T16:47:18.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.700+0000 7f494b400640 1 -- 172.21.15.28:0/1139238778 >> v2:172.21.15.28:3300/0 conn(0x7f494c1570a0 msgr2=0x7f494c070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:47:18.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.700+0000 7f494b400640 1 --2- 172.21.15.28:0/1139238778 >> v2:172.21.15.28:3300/0 conn(0x7f494c1570a0 0x7f494c070040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:18.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.700+0000 7f494b400640 1 -- 172.21.15.28:0/1139238778 >> v1:172.21.15.28:6789/0 conn(0x7f494c157480 legacy=0x7f494c06d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:47:18.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.700+0000 7f494b400640 1 -- 172.21.15.28:0/1139238778 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c15d600 con 0x7f494c0c6270 2024-09-13T16:47:18.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.700+0000 7f494b400640 1 -- 172.21.15.28:0/1139238778 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1654382301 0 0) 0x7f4938001670 con 0x7f494c0c6270 2024-09-13T16:47:18.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.700+0000 7f494b400640 1 -- 172.21.15.28:0/1139238778 >> v1:172.21.15.191:6789/0 conn(0x7f494c0c6270 legacy=0x7f494c0c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.700+0000 7f494b400640 1 --2- 172.21.15.28:0/1139238778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4934003970 0x7f4934003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.700+0000 7f494b400640 1 --2- 172.21.15.28:0/1139238778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49340047c0 0x7f4934004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.700+0000 7f494b400640 1 --2- 172.21.15.28:0/1139238778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4934005180 0x7f493400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.700+0000 7f494b400640 1 -- 172.21.15.28:0/1139238778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c15d320 con 0x7f4934005180 2024-09-13T16:47:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.700+0000 7f494b400640 1 -- 172.21.15.28:0/1139238778 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493400c0b0 con 0x7f49340047c0 2024-09-13T16:47:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.700+0000 7f494b400640 1 -- 172.21.15.28:0/1139238778 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f493400c270 con 0x7f4934003970 2024-09-13T16:47:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.700+0000 7f494be00640 1 --2- 172.21.15.28:0/1139238778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49340047c0 0x7f4934004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.700+0000 7f4950c00640 1 --2- 172.21.15.28:0/1139238778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4934003970 0x7f4934003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:18.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.701+0000 7f494be00640 1 -- 172.21.15.28:0/1139238778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4934003970 msgr2=0x7f4934003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:47:18.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.701+0000 7f494be00640 1 --2- 172.21.15.28:0/1139238778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4934003970 0x7f4934003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:18.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.701+0000 7f494be00640 1 -- 172.21.15.28:0/1139238778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4934005180 msgr2=0x7f493400b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:18.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.701+0000 7f494be00640 1 --2- 172.21.15.28:0/1139238778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4934005180 0x7f493400b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:18.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.701+0000 7f494be00640 1 -- 172.21.15.28:0/1139238778 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c15d600 con 0x7f49340047c0 2024-09-13T16:47:18.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.701+0000 7f4951600640 1 -- 172.21.15.28:0/1139238778 reap_dead start 2024-09-13T16:47:18.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.703+0000 7f494be00640 1 --2- 172.21.15.28:0/1139238778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49340047c0 0x7f4934004c40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f493800c260 tx=0x7f49340057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ed642c44e1c280 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:18.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.703+0000 7f494b400640 1 -- 172.21.15.28:0/1139238778 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4938005ad0 con 0x7f49340047c0 2024-09-13T16:47:18.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.703+0000 7f494b400640 1 -- 172.21.15.28:0/1139238778 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4938005c30 con 0x7f49340047c0 2024-09-13T16:47:18.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.703+0000 7f494b400640 1 -- 172.21.15.28:0/1139238778 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4938007070 con 0x7f49340047c0 2024-09-13T16:47:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.703+0000 7f4953600640 1 -- 172.21.15.28:0/1139238778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49340047c0 msgr2=0x7f4934004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.704+0000 7f4953600640 1 --2- 172.21.15.28:0/1139238778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49340047c0 0x7f4934004c40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f493800c260 tx=0x7f49340057f0 comp rx=0 tx=0).stop 2024-09-13T16:47:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.704+0000 7f4953600640 1 -- 172.21.15.28:0/1139238778 shutdown_connections 2024-09-13T16:47:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.704+0000 7f4953600640 1 --2- 172.21.15.28:0/1139238778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49340047c0 0x7f4934004c40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.704+0000 7f4953600640 1 -- 172.21.15.28:0/1139238778 >> 172.21.15.28:0/1139238778 conn(0x7f494c069e80 msgr2=0x7f494c06a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.704+0000 7f4953600640 1 -- 172.21.15.28:0/1139238778 shutdown_connections 2024-09-13T16:47:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.704+0000 7f4953600640 1 -- 172.21.15.28:0/1139238778 wait complete. 2024-09-13T16:47:18.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.706+0000 7f4953600640 1 Processor -- start 2024-09-13T16:47:18.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.706+0000 7f4953600640 1 -- start start 2024-09-13T16:47:18.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.707+0000 7f4953600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49340047c0 0x7f494c0c0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:18.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.707+0000 7f4953600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f494c1f29c0 0x7f494c1f56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:18.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.707+0000 7f4953600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f494c1f2e20 0x7f494c1f5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:18.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.707+0000 7f4953600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c15f8c0 con 0x7f494c1f2e20 2024-09-13T16:47:18.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.707+0000 7f4953600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c15fb10 con 0x7f49340047c0 2024-09-13T16:47:18.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.707+0000 7f4953600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494c15fdf0 con 0x7f494c1f29c0 2024-09-13T16:47:18.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.707+0000 7f4950c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49340047c0 0x7f494c0c0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:18.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.707+0000 7f494be00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f494c1f29c0 0x7f494c1f56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:18.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.707+0000 7f4951600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f494c1f2e20 0x7f494c1f5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:18.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.707+0000 7f4950c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49340047c0 0x7f494c0c0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:55328/0 (socket says 172.21.15.28:55328) 2024-09-13T16:47:18.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.707+0000 7f494be00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f494c1f29c0 0x7f494c1f56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:54910/0 (socket says 172.21.15.28:54910) 2024-09-13T16:47:18.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.707+0000 7f4950c00640 1 -- 172.21.15.28:0/707147955 learned_addr learned my addr 172.21.15.28:0/707147955 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:18.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.707+0000 7f4950c00640 1 -- 172.21.15.28:0/707147955 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f494c1f29c0 msgr2=0x7f494c1f56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:18.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.707+0000 7f4950c00640 1 --2- 172.21.15.28:0/707147955 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f494c1f29c0 0x7f494c1f56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:18.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.707+0000 7f4950c00640 1 -- 172.21.15.28:0/707147955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f494c1f2e20 msgr2=0x7f494c1f5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:18.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.707+0000 7f4950c00640 1 --2- 172.21.15.28:0/707147955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f494c1f2e20 0x7f494c1f5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:18.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.707+0000 7f4950c00640 1 -- 172.21.15.28:0/707147955 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c1600d0 con 0x7f49340047c0 2024-09-13T16:47:18.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.708+0000 7f4950c00640 1 --2- 172.21.15.28:0/707147955 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49340047c0 0x7f494c0c0b10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f494000b7f0 tx=0x7f494000bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:18.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.708+0000 7f4948c00640 1 -- 172.21.15.28:0/707147955 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4940010830 con 0x7f49340047c0 2024-09-13T16:47:18.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.708+0000 7f4948c00640 1 -- 172.21.15.28:0/707147955 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4940010990 con 0x7f49340047c0 2024-09-13T16:47:18.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.708+0000 7f4948c00640 1 -- 172.21.15.28:0/707147955 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4940010d20 con 0x7f49340047c0 2024-09-13T16:47:18.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.708+0000 7f4953600640 1 -- 172.21.15.28:0/707147955 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f494c1f21c0 con 0x7f49340047c0 2024-09-13T16:47:18.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.708+0000 7f4953600640 1 -- 172.21.15.28:0/707147955 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f494c157920 con 0x7f49340047c0 2024-09-13T16:47:18.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.711+0000 7f4948c00640 1 -- 172.21.15.28:0/707147955 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4940014430 con 0x7f49340047c0 2024-09-13T16:47:18.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.712+0000 7f4948c00640 1 --2- 172.21.15.28:0/707147955 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f492c06ac00 0x7f492c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:18.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.712+0000 7f4948c00640 1 -- 172.21.15.28:0/707147955 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4940089d60 con 0x7f49340047c0 2024-09-13T16:47:18.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.712+0000 7f494be00640 1 --2- 172.21.15.28:0/707147955 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f492c06ac00 0x7f492c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:18.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.713+0000 7f494be00640 1 --2- 172.21.15.28:0/707147955 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f492c06ac00 0x7f492c06d0c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4938033000 tx=0x7f493802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:18.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.715+0000 7f4953600640 1 -- 172.21.15.28:0/707147955 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f494c070040 con 0x7f49340047c0 2024-09-13T16:47:18.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:18.723+0000 7f4948c00640 1 -- 172.21.15.28:0/707147955 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4940022080 con 0x7f49340047c0 2024-09-13T16:47:19.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.199+0000 7f4953600640 1 -- 172.21.15.28:0/707147955 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f494c0c6890 con 0x7f49340047c0 2024-09-13T16:47:19.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.200+0000 7f4948c00640 1 -- 172.21.15.28:0/707147955 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f4940054ee0 con 0x7f49340047c0 2024-09-13T16:47:19.202 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:47:19.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.204+0000 7f4953600640 1 -- 172.21.15.28:0/707147955 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f492c06ac00 msgr2=0x7f492c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.204+0000 7f4953600640 1 --2- 172.21.15.28:0/707147955 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f492c06ac00 0x7f492c06d0c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4938033000 tx=0x7f493802e000 comp rx=0 tx=0).stop 2024-09-13T16:47:19.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.204+0000 7f4953600640 1 -- 172.21.15.28:0/707147955 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49340047c0 msgr2=0x7f494c0c0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.204+0000 7f4953600640 1 --2- 172.21.15.28:0/707147955 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49340047c0 0x7f494c0c0b10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f494000b7f0 tx=0x7f494000bcb0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.205+0000 7f4953600640 1 -- 172.21.15.28:0/707147955 shutdown_connections 2024-09-13T16:47:19.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.205+0000 7f4953600640 1 --2- 172.21.15.28:0/707147955 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f492c06ac00 0x7f492c06d0c0 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4938033000 tx=0x7f493802e000 comp rx=0 tx=0).stop 2024-09-13T16:47:19.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.205+0000 7f4953600640 1 --2- 172.21.15.28:0/707147955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f494c1f2e20 0x7f494c1f5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.205+0000 7f4953600640 1 --2- 172.21.15.28:0/707147955 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f494c1f29c0 0x7f494c1f56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.205+0000 7f4953600640 1 --2- 172.21.15.28:0/707147955 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49340047c0 0x7f494c0c0b10 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f494000b7f0 tx=0x7f494000bcb0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.205+0000 7f4953600640 1 -- 172.21.15.28:0/707147955 >> 172.21.15.28:0/707147955 conn(0x7f494c069e80 msgr2=0x7f494c15b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:19.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.212+0000 7f4953600640 1 -- 172.21.15.28:0/707147955 shutdown_connections 2024-09-13T16:47:19.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.216+0000 7f4953600640 1 -- 172.21.15.28:0/707147955 wait complete. 2024-09-13T16:47:19.241 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T16:47:19.241 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-13T16:47:19.241 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-13T16:47:19.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T16:47:19.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.502+0000 7f52af200640 1 Processor -- start 2024-09-13T16:47:19.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.502+0000 7f52af200640 1 -- start start 2024-09-13T16:47:19.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.503+0000 7f52af200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b00cd010 0x7f52b00c1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:19.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.503+0000 7f52af200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52b01573f0 0x7f52b00c7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:19.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.503+0000 7f52af200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b00c7550 con 0x7f52b00ccc30 2024-09-13T16:47:19.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.503+0000 7f52af200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b0159f10 con 0x7f52b01573f0 2024-09-13T16:47:19.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.503+0000 7f52af200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b015a1d0 con 0x7f52b00cd010 2024-09-13T16:47:19.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.503+0000 7f52ade00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f52b00ccc30 0x7f52b0156ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46866/0 (socket says 172.21.15.28:46866) 2024-09-13T16:47:19.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.503+0000 7f52ade00640 1 -- 172.21.15.28:0/1917509135 learned_addr learned my addr 172.21.15.28:0/1917509135 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:47:19.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.503+0000 7f52ae800640 1 --2- 172.21.15.28:0/1917509135 >> v2:172.21.15.191:3300/0 conn(0x7f52b01573f0 0x7f52b00c7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:19.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.503+0000 7f52ad400640 1 --2- 172.21.15.28:0/1917509135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b00cd010 0x7f52b00c1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:19.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.504+0000 7f52ae800640 1 -- 172.21.15.28:0/1917509135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b00cd010 msgr2=0x7f52b00c1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:47:19.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.504+0000 7f52ae800640 1 --2- 172.21.15.28:0/1917509135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b00cd010 0x7f52b00c1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.504+0000 7f52aca00640 1 -- 172.21.15.28:0/1917509135 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832329624 0 0) 0x7f52a0001110 con 0x7f52b00ccc30 2024-09-13T16:47:19.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.504+0000 7f52ae800640 1 -- 172.21.15.28:0/1917509135 >> v1:172.21.15.28:6789/0 conn(0x7f52b00ccc30 legacy=0x7f52b0156ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.504+0000 7f52ae800640 1 -- 172.21.15.28:0/1917509135 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b015a4b0 con 0x7f52b01573f0 2024-09-13T16:47:19.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.504+0000 7f52ad400640 1 --2- 172.21.15.28:0/1917509135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b00cd010 0x7f52b00c1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:47:19.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.505+0000 7f52ae800640 1 --2- 172.21.15.28:0/1917509135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b01573f0 0x7f52b00c7010 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f52a8009dc0 tx=0x7f52a802a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75c5abff32293768 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:19.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.505+0000 7f52aca00640 1 -- 172.21.15.28:0/1917509135 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a8007740 con 0x7f52b01573f0 2024-09-13T16:47:19.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.505+0000 7f52aca00640 1 -- 172.21.15.28:0/1917509135 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a802d050 con 0x7f52b01573f0 2024-09-13T16:47:19.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.505+0000 7f52aca00640 1 -- 172.21.15.28:0/1917509135 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a8007ac0 con 0x7f52b01573f0 2024-09-13T16:47:19.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.506+0000 7f52af200640 1 -- 172.21.15.28:0/1917509135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b01573f0 msgr2=0x7f52b00c7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.506+0000 7f52af200640 1 --2- 172.21.15.28:0/1917509135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b01573f0 0x7f52b00c7010 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f52a8009dc0 tx=0x7f52a802a680 comp rx=0 tx=0).stop 2024-09-13T16:47:19.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.506+0000 7f52af200640 1 -- 172.21.15.28:0/1917509135 shutdown_connections 2024-09-13T16:47:19.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.506+0000 7f52af200640 1 --2- 172.21.15.28:0/1917509135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b01573f0 0x7f52b00c7010 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.506+0000 7f52af200640 1 --2- 172.21.15.28:0/1917509135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b00cd010 0x7f52b00c1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.506+0000 7f52af200640 1 -- 172.21.15.28:0/1917509135 >> 172.21.15.28:0/1917509135 conn(0x7f52b006a5a0 msgr2=0x7f52b006ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:19.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.506+0000 7f52af200640 1 -- 172.21.15.28:0/1917509135 shutdown_connections 2024-09-13T16:47:19.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.506+0000 7f52af200640 1 -- 172.21.15.28:0/1917509135 wait complete. 2024-09-13T16:47:19.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.508+0000 7f52af200640 1 Processor -- start 2024-09-13T16:47:19.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.508+0000 7f52af200640 1 -- start start 2024-09-13T16:47:19.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.509+0000 7f52af200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b00ccc30 0x7f52b00af260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:19.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.509+0000 7f52af200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b00cd010 0x7f52b00cb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:19.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.509+0000 7f52af200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b01573f0 0x7f52b006c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:19.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.509+0000 7f52af200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b01e1270 con 0x7f52b01573f0 2024-09-13T16:47:19.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.509+0000 7f52af200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b01e0d10 con 0x7f52b00ccc30 2024-09-13T16:47:19.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.509+0000 7f52af200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b01e1770 con 0x7f52b00cd010 2024-09-13T16:47:19.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.509+0000 7f52ade00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b00ccc30 0x7f52b00af260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:19.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.509+0000 7f52ad400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b00cd010 0x7f52b00cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:19.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.509+0000 7f52ade00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b00ccc30 0x7f52b00af260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:37258/0 (socket says 172.21.15.28:37258) 2024-09-13T16:47:19.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.509+0000 7f52ade00640 1 -- 172.21.15.28:0/2660465671 learned_addr learned my addr 172.21.15.28:0/2660465671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:19.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.509+0000 7f52ae800640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b01573f0 0x7f52b006c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:19.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.510+0000 7f52ade00640 1 -- 172.21.15.28:0/2660465671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b00cd010 msgr2=0x7f52b00cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.510+0000 7f52ade00640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b00cd010 0x7f52b00cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.510+0000 7f52ade00640 1 -- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b01573f0 msgr2=0x7f52b006c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.510+0000 7f52ade00640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b01573f0 0x7f52b006c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.510+0000 7f52ade00640 1 -- 172.21.15.28:0/2660465671 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529c009400 con 0x7f52b00ccc30 2024-09-13T16:47:19.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.510+0000 7f52ae800640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b01573f0 0x7f52b006c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:47:19.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.510+0000 7f52ad400640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52b00cd010 0x7f52b00cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:47:19.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.510+0000 7f52ade00640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b00ccc30 0x7f52b00af260 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f52a000c7b0 tx=0x7f52a000cc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:19.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.510+0000 7f52a6000640 1 -- 172.21.15.28:0/2660465671 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a00072d0 con 0x7f52b00ccc30 2024-09-13T16:47:19.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.510+0000 7f52a6000640 1 -- 172.21.15.28:0/2660465671 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a0007450 con 0x7f52b00ccc30 2024-09-13T16:47:19.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.510+0000 7f52af200640 1 -- 172.21.15.28:0/2660465671 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a8009a70 con 0x7f52b00ccc30 2024-09-13T16:47:19.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.510+0000 7f52a6000640 1 -- 172.21.15.28:0/2660465671 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a00047e0 con 0x7f52b00ccc30 2024-09-13T16:47:19.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.510+0000 7f52af200640 1 -- 172.21.15.28:0/2660465671 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b01e0f90 con 0x7f52b00ccc30 2024-09-13T16:47:19.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.511+0000 7f52af200640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b014efb0 0x7f52b006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:19.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.511+0000 7f52af200640 1 -- 172.21.15.28:0/2660465671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b006ab50 con 0x7f52b014efb0 2024-09-13T16:47:19.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.512+0000 7f52ad400640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b014efb0 0x7f52b006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:19.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.512+0000 7f52ad400640 1 -- 172.21.15.28:0/2660465671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52b015aa70 con 0x7f52b014efb0 2024-09-13T16:47:19.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.512+0000 7f52a6000640 1 -- 172.21.15.28:0/2660465671 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f52a0004960 con 0x7f52b00ccc30 2024-09-13T16:47:19.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.512+0000 7f52ad400640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b014efb0 0x7f52b006c7e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f52b00b1f40 tx=0x7f529c00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:19.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.513+0000 7f52a6000640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f529006ac30 0x7f529006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:19.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.513+0000 7f52a6000640 1 -- 172.21.15.28:0/2660465671 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f52a00886b0 con 0x7f52b00ccc30 2024-09-13T16:47:19.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.513+0000 7f52ae800640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f529006ac30 0x7f529006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:19.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.513+0000 7f52a6000640 1 -- 172.21.15.28:0/2660465671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c014440 con 0x7f52b014efb0 2024-09-13T16:47:19.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.513+0000 7f52a6000640 1 -- 172.21.15.28:0/2660465671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f529c0145a0 con 0x7f52b014efb0 2024-09-13T16:47:19.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.513+0000 7f52a6000640 1 -- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b014efb0 msgr2=0x7f52b006c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.513+0000 7f52a6000640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b014efb0 0x7f52b006c7e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f52b00b1f40 tx=0x7f529c00f610 comp rx=0 tx=0).stop 2024-09-13T16:47:19.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.513+0000 7f52a6000640 1 -- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b014efb0 msgr2=0x7f52b006c7e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:47:19.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.513+0000 7f52a6000640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b014efb0 0x7f52b006c7e0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f52b00b1f40 tx=0x7f529c00f610 comp rx=0 tx=0).stop 2024-09-13T16:47:19.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.514+0000 7f52ae800640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f529006ac30 0x7f529006d0f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f52a8034000 tx=0x7f52a8033040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:19.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.520+0000 7f52af200640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b0142960 0x7f52b0142d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:19.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.520+0000 7f52af200640 1 -- 172.21.15.28:0/2660465671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b00c7550 con 0x7f52b0142960 2024-09-13T16:47:19.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.521+0000 7f52ad400640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b0142960 0x7f52b0142d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.521+0000 7f52ad400640 1 -- 172.21.15.28:0/2660465671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f52b00c1630 con 0x7f52b0142960 2024-09-13T16:47:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.522+0000 7f52ad400640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b0142960 0x7f52b0142d70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f529c01c000 tx=0x7f529c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.522+0000 7f52a6000640 1 -- 172.21.15.28:0/2660465671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c021040 con 0x7f52b0142960 2024-09-13T16:47:19.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.522+0000 7f52a6000640 1 -- 172.21.15.28:0/2660465671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4460 (secure 0 0 0) 0x7f529c004220 con 0x7f52b0142960 2024-09-13T16:47:19.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.522+0000 7f52a6000640 1 -- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b0142960 msgr2=0x7f52b0142d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.522+0000 7f52a6000640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b0142960 0x7f52b0142d70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f529c01c000 tx=0x7f529c0046c0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.522+0000 7f52a6000640 1 -- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b0142960 msgr2=0x7f52b0142d70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:47:19.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.522+0000 7f52a6000640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b0142960 0x7f52b0142d70 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f529c01c000 tx=0x7f529c0046c0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.524 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:47:19.524 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:47:19.524 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:47:19.524 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "leader", 2024-09-13T16:47:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 22, 2024-09-13T16:47:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:47:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:47:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:47:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:47:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 14, 2024-09-13T16:47:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:47:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:47:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:47:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:19.525 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:47:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:47:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:19.526 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:19.527 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:47:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:47:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:47:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:47:19.528 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:47:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:47:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:47:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:47:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:47:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:47:19.529 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:47:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:47:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:47:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:47:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:47:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:47:19.530 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:47:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:19.531 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:47:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:47:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.533 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:47:19.534 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:47:19.534 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:19.534 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:19.534 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.534 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:19.535 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:47:19.535 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:19.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.535 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:19.535 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:47:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:47:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:47:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:19.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:47:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:47:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:19.538 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:47:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:47:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:19.540 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:47:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:47:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:47:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:47:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:47:19.542 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:47:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.543 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:47:19.544 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:47:19.544 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:19.544 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:19.544 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:19.544 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:47:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:47:19.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:47:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:19.546 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:47:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:47:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.547 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:47:19.547 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:47:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.523+0000 7f52af200640 1 -- 172.21.15.28:0/2660465671 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f529006ac30 msgr2=0x7f529006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.523+0000 7f52af200640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f529006ac30 0x7f529006d0f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f52a8034000 tx=0x7f52a8033040 comp rx=0 tx=0).stop 2024-09-13T16:47:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.523+0000 7f52af200640 1 -- 172.21.15.28:0/2660465671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b00ccc30 msgr2=0x7f52b00af260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.523+0000 7f52af200640 1 --2- 172.21.15.28:0/2660465671 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52b00ccc30 0x7f52b00af260 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f52a000c7b0 tx=0x7f52a000cc70 comp rx=0 tx=0).stop 2024-09-13T16:47:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.523+0000 7f52ae800640 1 -- 172.21.15.28:0/2660465671 reap_dead start 2024-09-13T16:47:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.523+0000 7f52ae800640 1 -- 172.21.15.28:0/2660465671 reap_dead start 2024-09-13T16:47:19.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.523+0000 7f52af200640 1 -- 172.21.15.28:0/2660465671 shutdown_connections 2024-09-13T16:47:19.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.523+0000 7f52af200640 1 -- 172.21.15.28:0/2660465671 >> 172.21.15.28:0/2660465671 conn(0x7f52b006a5a0 msgr2=0x7f52b0156ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:19.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.523+0000 7f52af200640 1 -- 172.21.15.28:0/2660465671 shutdown_connections 2024-09-13T16:47:19.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.523+0000 7f52af200640 1 -- 172.21.15.28:0/2660465671 wait complete. 2024-09-13T16:47:19.550 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:47:19.550 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-13T16:47:19.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T16:47:19.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.772+0000 7f78ad800640 1 Processor -- start 2024-09-13T16:47:19.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.772+0000 7f78ad800640 1 -- start start 2024-09-13T16:47:19.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.774+0000 7f78ad800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f78a8152e50 0x7f78a80caec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:19.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.774+0000 7f78ad800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78a80ca420 0x7f78a80cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:19.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.774+0000 7f78ad800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a8153230 con 0x7f78a80c9230 2024-09-13T16:47:19.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.774+0000 7f78ad800640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a806aa90 con 0x7f78a8152e50 2024-09-13T16:47:19.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.774+0000 7f78ad800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a8158a80 con 0x7f78a80ca420 2024-09-13T16:47:19.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.775+0000 7f78a7400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78a80ca420 0x7f78a80cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:19.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.775+0000 7f78a7e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f78a8152e50 0x7f78a80caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:19.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.775+0000 7f78a7e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f78a8152e50 0x7f78a80caec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:37264/0 (socket says 172.21.15.28:37264) 2024-09-13T16:47:19.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.775+0000 7f78a7400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78a80ca420 0x7f78a80cb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:43062/0 (socket says 172.21.15.28:43062) 2024-09-13T16:47:19.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.775+0000 7f78a7e00640 1 -- 172.21.15.28:0/1558872633 learned_addr learned my addr 172.21.15.28:0/1558872633 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:19.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.775+0000 7f78ace00640 1 --1- 172.21.15.28:0/1558872633 >> v1:172.21.15.28:6789/0 conn(0x7f78a80c9230 0x7f78a806aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46872/0 (socket says 172.21.15.28:46872) 2024-09-13T16:47:19.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.775+0000 7f78a7e00640 1 -- 172.21.15.28:0/1558872633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78a80ca420 msgr2=0x7f78a80cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:47:19.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.775+0000 7f78a7e00640 1 --2- 172.21.15.28:0/1558872633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78a80ca420 0x7f78a80cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.775+0000 7f78a7e00640 1 -- 172.21.15.28:0/1558872633 >> v1:172.21.15.28:6789/0 conn(0x7f78a80c9230 legacy=0x7f78a806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.775+0000 7f78a7e00640 1 -- 172.21.15.28:0/1558872633 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a8158c90 con 0x7f78a8152e50 2024-09-13T16:47:19.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.775+0000 7f78a7400640 1 --2- 172.21.15.28:0/1558872633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78a80ca420 0x7f78a80cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:47:19.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.776+0000 7f78a7e00640 1 --2- 172.21.15.28:0/1558872633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a8152e50 0x7f78a80caec0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7898009870 tx=0x7f789802a2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0a2ebebedd4b898 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:19.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.776+0000 7f78a6a00640 1 -- 172.21.15.28:0/1558872633 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7898032640 con 0x7f78a8152e50 2024-09-13T16:47:19.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.776+0000 7f78a6a00640 1 -- 172.21.15.28:0/1558872633 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78980327a0 con 0x7f78a8152e50 2024-09-13T16:47:19.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.776+0000 7f78a6a00640 1 -- 172.21.15.28:0/1558872633 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7898032b30 con 0x7f78a8152e50 2024-09-13T16:47:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.777+0000 7f78ad800640 1 -- 172.21.15.28:0/1558872633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a8152e50 msgr2=0x7f78a80caec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.777+0000 7f78ad800640 1 --2- 172.21.15.28:0/1558872633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a8152e50 0x7f78a80caec0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7898009870 tx=0x7f789802a2f0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.777+0000 7f78ad800640 1 -- 172.21.15.28:0/1558872633 shutdown_connections 2024-09-13T16:47:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.777+0000 7f78ad800640 1 --2- 172.21.15.28:0/1558872633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78a80ca420 0x7f78a80cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.777+0000 7f78ad800640 1 --2- 172.21.15.28:0/1558872633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a8152e50 0x7f78a80caec0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.777+0000 7f78ad800640 1 -- 172.21.15.28:0/1558872633 >> 172.21.15.28:0/1558872633 conn(0x7f78a8065c50 msgr2=0x7f78a8066060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:19.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.777+0000 7f78ad800640 1 -- 172.21.15.28:0/1558872633 shutdown_connections 2024-09-13T16:47:19.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.777+0000 7f78ad800640 1 -- 172.21.15.28:0/1558872633 wait complete. 2024-09-13T16:47:19.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.779+0000 7f78ad800640 1 Processor -- start 2024-09-13T16:47:19.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.779+0000 7f78ad800640 1 -- start start 2024-09-13T16:47:19.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.780+0000 7f78ad800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78a80c9230 0x7f78a8155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:19.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.780+0000 7f78ad800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a80ca420 0x7f78a8154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:19.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.780+0000 7f78ad800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78a8152e50 0x7f78a814e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:19.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.780+0000 7f78ad800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a81ee890 con 0x7f78a8152e50 2024-09-13T16:47:19.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.780+0000 7f78ad800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a81eeb70 con 0x7f78a80ca420 2024-09-13T16:47:19.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.780+0000 7f78ad800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a81eee50 con 0x7f78a80c9230 2024-09-13T16:47:19.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.780+0000 7f78a7e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78a80c9230 0x7f78a8155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:19.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.780+0000 7f78ace00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78a8152e50 0x7f78a814e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:19.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.780+0000 7f78a7400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a80ca420 0x7f78a8154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:19.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.780+0000 7f78a7e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78a80c9230 0x7f78a8155630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:43066/0 (socket says 172.21.15.28:43066) 2024-09-13T16:47:19.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.780+0000 7f78a7e00640 1 -- 172.21.15.28:0/3245980379 learned_addr learned my addr 172.21.15.28:0/3245980379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:19.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.781+0000 7f78a7e00640 1 -- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a80ca420 msgr2=0x7f78a8154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.781+0000 7f78a7e00640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a80ca420 0x7f78a8154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.781+0000 7f78a7e00640 1 -- 172.21.15.28:0/3245980379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78a8152e50 msgr2=0x7f78a814e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.781+0000 7f78a7e00640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78a8152e50 0x7f78a814e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.781+0000 7f78a7e00640 1 -- 172.21.15.28:0/3245980379 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7894009400 con 0x7f78a80c9230 2024-09-13T16:47:19.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.781+0000 7f78a7400640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a80ca420 0x7f78a8154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:47:19.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.781+0000 7f78a7e00640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78a80c9230 0x7f78a8155630 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7898009870 tx=0x7f78980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:19.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.781+0000 7f789fe00640 1 -- 172.21.15.28:0/3245980379 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789803b810 con 0x7f78a80c9230 2024-09-13T16:47:19.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.781+0000 7f789fe00640 1 -- 172.21.15.28:0/3245980379 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789803b970 con 0x7f78a80c9230 2024-09-13T16:47:19.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.781+0000 7f789fe00640 1 -- 172.21.15.28:0/3245980379 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789803bd20 con 0x7f78a80c9230 2024-09-13T16:47:19.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.781+0000 7f78ad800640 1 -- 172.21.15.28:0/3245980379 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78980094d0 con 0x7f78a80c9230 2024-09-13T16:47:19.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.781+0000 7f78ad800640 1 -- 172.21.15.28:0/3245980379 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a8156170 con 0x7f78a80c9230 2024-09-13T16:47:19.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.783+0000 7f78ad800640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a80c8300 0x7f78a814e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:19.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.783+0000 7f78ad800640 1 -- 172.21.15.28:0/3245980379 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a80720f0 con 0x7f78a80c8300 2024-09-13T16:47:19.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.783+0000 7f78a7400640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a80c8300 0x7f78a814e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:19.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.783+0000 7f78a7400640 1 -- 172.21.15.28:0/3245980379 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78a80c4eb0 con 0x7f78a80c8300 2024-09-13T16:47:19.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.783+0000 7f78a7400640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a80c8300 0x7f78a814e960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f78a8150aa0 tx=0x7f789400f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:19.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.784+0000 7f789fe00640 1 -- 172.21.15.28:0/3245980379 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7894004020 con 0x7f78a80c8300 2024-09-13T16:47:19.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.784+0000 7f789fe00640 1 -- 172.21.15.28:0/3245980379 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f7894004180 con 0x7f78a80c8300 2024-09-13T16:47:19.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.784+0000 7f789fe00640 1 -- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a80c8300 msgr2=0x7f78a814e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.784+0000 7f789fe00640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a80c8300 0x7f78a814e960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f78a8150aa0 tx=0x7f789400f610 comp rx=0 tx=0).stop 2024-09-13T16:47:19.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.784+0000 7f789fe00640 1 -- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a80c8300 msgr2=0x7f78a814e960 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:47:19.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.784+0000 7f789fe00640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a80c8300 0x7f78a814e960 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f78a8150aa0 tx=0x7f789400f610 comp rx=0 tx=0).stop 2024-09-13T16:47:19.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.785+0000 7f789fe00640 1 -- 172.21.15.28:0/3245980379 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7898039040 con 0x7f78a80c9230 2024-09-13T16:47:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.786+0000 7f789fe00640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f788806a980 0x7f788806ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.787+0000 7f789fe00640 1 -- 172.21.15.28:0/3245980379 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f789802ae40 con 0x7f78a80c9230 2024-09-13T16:47:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.787+0000 7f78a7400640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f788806a980 0x7f788806ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:19.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.787+0000 7f78a7400640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f788806a980 0x7f788806ce40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f789401b000 tx=0x7f7894008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.791+0000 7f78ad800640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a81f1ab0 0x7f78a81f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:19.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.791+0000 7f78ad800640 1 -- 172.21.15.28:0/3245980379 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a8159530 con 0x7f78a81f1ab0 2024-09-13T16:47:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.791+0000 7f78ace00640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a81f1ab0 0x7f78a81f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:19.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.792+0000 7f78ace00640 1 -- 172.21.15.28:0/3245980379 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f78a8159250 con 0x7f78a81f1ab0 2024-09-13T16:47:19.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.792+0000 7f78ace00640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a81f1ab0 0x7f78a81f1ec0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f78a0007300 tx=0x7f78a00077c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:19.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.792+0000 7f789fe00640 1 -- 172.21.15.28:0/3245980379 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a0017070 con 0x7f78a81f1ab0 2024-09-13T16:47:19.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.793+0000 7f789fe00640 1 -- 172.21.15.28:0/3245980379 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4790 (secure 0 0 0) 0x7f78a0007a10 con 0x7f78a81f1ab0 2024-09-13T16:47:19.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.793+0000 7f789fe00640 1 -- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a81f1ab0 msgr2=0x7f78a81f1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.793+0000 7f789fe00640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a81f1ab0 0x7f78a81f1ec0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f78a0007300 tx=0x7f78a00077c0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.793+0000 7f789fe00640 1 -- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a81f1ab0 msgr2=0x7f78a81f1ec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:47:19.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.793+0000 7f789fe00640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78a81f1ab0 0x7f78a81f1ec0 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f78a0007300 tx=0x7f78a00077c0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.795 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:47:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:47:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:47:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:47:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 22, 2024-09-13T16:47:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:47:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:47:19.795 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:47:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:47:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 14, 2024-09-13T16:47:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:47:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:47:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:47:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:19.796 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:47:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:47:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:19.797 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:47:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:47:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:47:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:47:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:47:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:47:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:47:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:47:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:47:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:47:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:47:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:47:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:47:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:47:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:47:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:47:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:47:19.799 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:47:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:47:19.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:47:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:47:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:47:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:19.801 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:47:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:47:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:47:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:47:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:47:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:47:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:47:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:19.803 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:47:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:47:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:47:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:47:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:19.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:47:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:47:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:47:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:47:19.805 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:47:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:47:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:47:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:47:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:47:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:47:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:47:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:19.807 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:47:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:47:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:19.808 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:47:19.808 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:47:19.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.793+0000 7f78ad800640 1 -- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f788806a980 msgr2=0x7f788806ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.793+0000 7f78ad800640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f788806a980 0x7f788806ce40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f789401b000 tx=0x7f7894008030 comp rx=0 tx=0).stop 2024-09-13T16:47:19.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.793+0000 7f78ad800640 1 -- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78a80c9230 msgr2=0x7f78a8155630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:19.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.793+0000 7f78ad800640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78a80c9230 0x7f78a8155630 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7898009870 tx=0x7f78980046c0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.793+0000 7f78ace00640 1 -- 172.21.15.28:0/3245980379 reap_dead start 2024-09-13T16:47:19.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.793+0000 7f78ad800640 1 -- 172.21.15.28:0/3245980379 shutdown_connections 2024-09-13T16:47:19.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.793+0000 7f78ad800640 1 --2- 172.21.15.28:0/3245980379 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f78a80c9230 0x7f78a8155630 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:19.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.793+0000 7f78ad800640 1 -- 172.21.15.28:0/3245980379 >> 172.21.15.28:0/3245980379 conn(0x7f78a8065c50 msgr2=0x7f78a8151ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:19.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.793+0000 7f78ad800640 1 -- 172.21.15.28:0/3245980379 shutdown_connections 2024-09-13T16:47:19.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:19.793+0000 7f78ad800640 1 -- 172.21.15.28:0/3245980379 wait complete. 2024-09-13T16:47:19.810 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:47:19.811 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-13T16:47:19.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T16:47:20.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.066+0000 7f84f1e00640 1 Processor -- start 2024-09-13T16:47:20.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.067+0000 7f84f1e00640 1 -- start start 2024-09-13T16:47:20.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.068+0000 7f84f1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f84ec152d50 0x7f84ec0c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:20.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.068+0000 7f84f1e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f84ec0c5340 0x7f84ec06d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:20.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.068+0000 7f84f1e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84ec0c6cf0 con 0x7f84ec0c5770 2024-09-13T16:47:20.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.068+0000 7f84f1e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84ec15c080 con 0x7f84ec152d50 2024-09-13T16:47:20.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.068+0000 7f84f1e00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84ec15cbc0 con 0x7f84ec0c5340 2024-09-13T16:47:20.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.068+0000 7f84ea000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f84ec0c5340 0x7f84ec06d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:20.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.069+0000 7f84ea000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f84ec0c5340 0x7f84ec06d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:37304/0 (socket says 172.21.15.28:37304) 2024-09-13T16:47:20.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.069+0000 7f84ea000640 1 -- 172.21.15.28:0/3714598544 learned_addr learned my addr 172.21.15.28:0/3714598544 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:20.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.069+0000 7f84ea000640 1 -- 172.21.15.28:0/3714598544 >> v2:172.21.15.28:3300/0 conn(0x7f84ec152d50 msgr2=0x7f84ec0c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:47:20.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.069+0000 7f84ea000640 1 --2- 172.21.15.28:0/3714598544 >> v2:172.21.15.28:3300/0 conn(0x7f84ec152d50 0x7f84ec0c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:20.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.069+0000 7f84ea000640 1 -- 172.21.15.28:0/3714598544 >> v1:172.21.15.28:6789/0 conn(0x7f84ec0c5770 legacy=0x7f84ec06fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:47:20.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.070+0000 7f84ea000640 1 -- 172.21.15.28:0/3714598544 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ec0c5d10 con 0x7f84ec0c5340 2024-09-13T16:47:20.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.071+0000 7f84ea000640 1 --2- 172.21.15.28:0/3714598544 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84ec0c5340 0x7f84ec06d230 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f84ec0c8380 tx=0x7f84d802a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e83e4b5f02af837 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:20.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.071+0000 7f84e9600640 1 -- 172.21.15.28:0/3714598544 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d8032640 con 0x7f84ec0c5340 2024-09-13T16:47:20.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.071+0000 7f84e9600640 1 -- 172.21.15.28:0/3714598544 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d80327a0 con 0x7f84ec0c5340 2024-09-13T16:47:20.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.071+0000 7f84e9600640 1 -- 172.21.15.28:0/3714598544 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d8032b30 con 0x7f84ec0c5340 2024-09-13T16:47:20.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.073+0000 7f84f1e00640 1 -- 172.21.15.28:0/3714598544 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84ec0c5340 msgr2=0x7f84ec06d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:20.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.073+0000 7f84f1e00640 1 --2- 172.21.15.28:0/3714598544 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84ec0c5340 0x7f84ec06d230 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f84ec0c8380 tx=0x7f84d802a330 comp rx=0 tx=0).stop 2024-09-13T16:47:20.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.074+0000 7f84f1e00640 1 -- 172.21.15.28:0/3714598544 shutdown_connections 2024-09-13T16:47:20.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.074+0000 7f84f1e00640 1 --2- 172.21.15.28:0/3714598544 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84ec0c5340 0x7f84ec06d230 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:20.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.074+0000 7f84f1e00640 1 --2- 172.21.15.28:0/3714598544 >> v2:172.21.15.28:3300/0 conn(0x7f84ec152d50 0x7f84ec0c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:20.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.074+0000 7f84f1e00640 1 -- 172.21.15.28:0/3714598544 >> 172.21.15.28:0/3714598544 conn(0x7f84ec0ca6b0 msgr2=0x7f84ec0caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:20.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.075+0000 7f84f1e00640 1 -- 172.21.15.28:0/3714598544 shutdown_connections 2024-09-13T16:47:20.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.076+0000 7f84f1e00640 1 -- 172.21.15.28:0/3714598544 wait complete. 2024-09-13T16:47:20.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.080+0000 7f84f1e00640 1 Processor -- start 2024-09-13T16:47:20.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.080+0000 7f84f1e00640 1 -- start start 2024-09-13T16:47:20.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.082+0000 7f84f1e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84ec0c5340 0x7f84ec149f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:20.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.083+0000 7f84f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec0c5770 0x7f84ec14f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:20.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.083+0000 7f84f1e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84ec152d50 0x7f84ec14fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:20.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.084+0000 7f84f1e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec155cf0 con 0x7f84ec0c5770 2024-09-13T16:47:20.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.084+0000 7f84f1e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec1ef670 con 0x7f84ec0c5340 2024-09-13T16:47:20.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.084+0000 7f84eaa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84ec0c5340 0x7f84ec149f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:20.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.084+0000 7f84eaa00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84ec0c5340 0x7f84ec149f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:37312/0 (socket says 172.21.15.28:37312) 2024-09-13T16:47:20.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.084+0000 7f84eaa00640 1 -- 172.21.15.28:0/3001119882 learned_addr learned my addr 172.21.15.28:0/3001119882 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:20.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.084+0000 7f84ea000640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec0c5770 0x7f84ec14f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:20.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.084+0000 7f84f1e00640 1 -- 172.21.15.28:0/3001119882 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec154e00 con 0x7f84ec152d50 2024-09-13T16:47:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.085+0000 7f84eb400640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84ec152d50 0x7f84ec14fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.085+0000 7f84eaa00640 1 -- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84ec152d50 msgr2=0x7f84ec14fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.085+0000 7f84eaa00640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84ec152d50 0x7f84ec14fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.085+0000 7f84eaa00640 1 -- 172.21.15.28:0/3001119882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec0c5770 msgr2=0x7f84ec14f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.085+0000 7f84eaa00640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec0c5770 0x7f84ec14f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.085+0000 7f84eaa00640 1 -- 172.21.15.28:0/3001119882 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d80094d0 con 0x7f84ec0c5340 2024-09-13T16:47:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.085+0000 7f84ea000640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84ec0c5770 0x7f84ec14f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:47:20.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.086+0000 7f84eaa00640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84ec0c5340 0x7f84ec149f60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00b410 tx=0x7f84dc00b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:20.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.086+0000 7f84e3400640 1 -- 172.21.15.28:0/3001119882 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc014e90 con 0x7f84ec0c5340 2024-09-13T16:47:20.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.086+0000 7f84e3400640 1 -- 172.21.15.28:0/3001119882 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84dc01c440 con 0x7f84ec0c5340 2024-09-13T16:47:20.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.086+0000 7f84e3400640 1 -- 172.21.15.28:0/3001119882 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc01c7d0 con 0x7f84ec0c5340 2024-09-13T16:47:20.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.086+0000 7f84f1e00640 1 -- 172.21.15.28:0/3001119882 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84ec1550e0 con 0x7f84ec0c5340 2024-09-13T16:47:20.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.087+0000 7f84f1e00640 1 -- 172.21.15.28:0/3001119882 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84ec1556a0 con 0x7f84ec0c5340 2024-09-13T16:47:20.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.089+0000 7f84f1e00640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84c4002a30 0x7f84c4004ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:20.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.089+0000 7f84f1e00640 1 -- 172.21.15.28:0/3001119882 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c4007e80 con 0x7f84c4002a30 2024-09-13T16:47:20.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.090+0000 7f84ea000640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84c4002a30 0x7f84c4004ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:20.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.090+0000 7f84ea000640 1 -- 172.21.15.28:0/3001119882 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f84c4008040 con 0x7f84c4002a30 2024-09-13T16:47:20.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.091+0000 7f84ea000640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84c4002a30 0x7f84c4004ef0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f84ec156720 tx=0x7f84d80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:20.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.091+0000 7f84e3400640 1 -- 172.21.15.28:0/3001119882 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f84dc0253b0 con 0x7f84ec0c5340 2024-09-13T16:47:20.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.093+0000 7f84e3400640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f84cc06ac80 0x7f84cc06d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:20.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.093+0000 7f84e3400640 1 -- 172.21.15.28:0/3001119882 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f84dc091240 con 0x7f84ec0c5340 2024-09-13T16:47:20.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.094+0000 7f84e3400640 1 -- 172.21.15.28:0/3001119882 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d8032640 con 0x7f84c4002a30 2024-09-13T16:47:20.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.094+0000 7f84e3400640 1 -- 172.21.15.28:0/3001119882 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f84d8040070 con 0x7f84c4002a30 2024-09-13T16:47:20.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.094+0000 7f84e3400640 1 -- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84c4002a30 msgr2=0x7f84c4004ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:20.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.094+0000 7f84e3400640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84c4002a30 0x7f84c4004ef0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f84ec156720 tx=0x7f84d80046c0 comp rx=0 tx=0).stop 2024-09-13T16:47:20.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.094+0000 7f84eb400640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f84cc06ac80 0x7f84cc06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:20.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.094+0000 7f84e3400640 1 -- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84c4002a30 msgr2=0x7f84c4004ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:47:20.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.094+0000 7f84e3400640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84c4002a30 0x7f84c4004ef0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:20.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.094+0000 7f84eb400640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f84cc06ac80 0x7f84cc06d140 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f84e40012e0 tx=0x7f84e4009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.108+0000 7f84f1e00640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84c4006e70 0x7f84c40072c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.108+0000 7f84f1e00640 1 -- 172.21.15.28:0/3001119882 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c400a3e0 con 0x7f84c4006e70 2024-09-13T16:47:20.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.108+0000 7f84ea000640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84c4006e70 0x7f84c40072c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:20.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.109+0000 7f84ea000640 1 -- 172.21.15.28:0/3001119882 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f84c4007800 con 0x7f84c4006e70 2024-09-13T16:47:20.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.109+0000 7f84ea000640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84c4006e70 0x7f84c40072c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f84d802a370 tx=0x7f84d8008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.110+0000 7f84e3400640 1 -- 172.21.15.28:0/3001119882 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d804da90 con 0x7f84c4006e70 2024-09-13T16:47:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.110+0000 7f84e3400640 1 -- 172.21.15.28:0/3001119882 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+5099 (secure 0 0 0) 0x7f84d804dbf0 con 0x7f84c4006e70 2024-09-13T16:47:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.110+0000 7f84e3400640 1 -- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84c4006e70 msgr2=0x7f84c40072c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.110+0000 7f84e3400640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84c4006e70 0x7f84c40072c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f84d802a370 tx=0x7f84d8008030 comp rx=0 tx=0).stop 2024-09-13T16:47:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.110+0000 7f84e3400640 1 -- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84c4006e70 msgr2=0x7f84c40072c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:47:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.110+0000 7f84e3400640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84c4006e70 0x7f84c40072c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:20.112 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:47:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:47:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:47:20.112 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:47:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 22, 2024-09-13T16:47:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:47:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:47:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:47:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:47:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 15, 2024-09-13T16:47:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:47:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:47:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:47:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:20.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:20.114 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:47:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:47:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:20.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:47:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:47:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:47:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:47:20.117 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:47:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:47:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:47:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:47:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:47:20.118 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:47:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:47:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:47:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:47:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:47:20.119 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:47:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:47:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:47:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:20.121 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:47:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:20.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:47:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:47:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:47:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:47:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:20.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:47:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:20.125 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:47:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:47:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:20.126 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:47:20.127 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:47:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:47:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:20.129 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:47:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:20.130 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:47:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:47:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:20.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:47:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:47:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:20.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:47:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:20.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:47:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:20.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:47:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:47:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:47:20.136 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:47:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:47:20.137 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:47:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:20.138 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 5 2024-09-13T16:47:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:47:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 6 2024-09-13T16:47:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:47:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:20.139 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x2f018fb87aa4aafe", 2024-09-13T16:47:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "luminous", 2024-09-13T16:47:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:47:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4 2024-09-13T16:47:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": [ 2024-09-13T16:47:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:20.140 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:47:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:20.141 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:47:20.141 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:47:20.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.111+0000 7f84f1e00640 1 -- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f84cc06ac80 msgr2=0x7f84cc06d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:20.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.111+0000 7f84f1e00640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f84cc06ac80 0x7f84cc06d140 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f84e40012e0 tx=0x7f84e4009040 comp rx=0 tx=0).stop 2024-09-13T16:47:20.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.111+0000 7f84f1e00640 1 -- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84ec0c5340 msgr2=0x7f84ec149f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.111+0000 7f84f1e00640 1 --2- 172.21.15.28:0/3001119882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84ec0c5340 0x7f84ec149f60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00b410 tx=0x7f84dc00b8d0 comp rx=0 tx=0).stop 2024-09-13T16:47:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.111+0000 7f84eb400640 1 -- 172.21.15.28:0/3001119882 reap_dead start 2024-09-13T16:47:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.111+0000 7f84eb400640 1 -- 172.21.15.28:0/3001119882 reap_dead start 2024-09-13T16:47:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.111+0000 7f84f1e00640 1 -- 172.21.15.28:0/3001119882 shutdown_connections 2024-09-13T16:47:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.111+0000 7f84f1e00640 1 -- 172.21.15.28:0/3001119882 >> 172.21.15.28:0/3001119882 conn(0x7f84ec0ca6b0 msgr2=0x7f84ec0c9320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:20.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.111+0000 7f84f1e00640 1 -- 172.21.15.28:0/3001119882 shutdown_connections 2024-09-13T16:47:20.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.111+0000 7f84f1e00640 1 -- 172.21.15.28:0/3001119882 wait complete. 2024-09-13T16:47:20.146 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:47:20.146 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-13T16:47:20.146 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-13T16:47:20.146 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-13T16:47:20.147 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-13T16:47:20.147 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-13T16:47:20.147 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:47:20.174 INFO:tasks.ceph.mon.a:Stopped 2024-09-13T16:47:20.175 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-13T16:47:20.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:47:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.530+0000 7f519ee00640 1 Processor -- start 2024-09-13T16:47:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.530+0000 7f519ee00640 1 -- start start 2024-09-13T16:47:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.532+0000 7f519ee00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f51981570a0 0x7f5198070040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.532+0000 7f519ee00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f51980c6270 0x7f51980c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.532+0000 7f519ee00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519815df30 con 0x7f5198157480 2024-09-13T16:47:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.532+0000 7f519ee00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5198161340 con 0x7f51981570a0 2024-09-13T16:47:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.532+0000 7f519ee00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f519815d320 con 0x7f51980c6270 2024-09-13T16:47:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.532+0000 7f5197e00640 1 -- >> v2:172.21.15.28:3300/0 conn(0x7f51981570a0 msgr2=0x7f5198070040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:47:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.533+0000 7f5197e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f51981570a0 0x7f5198070040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.533+0000 7f519ce00640 1 -- >> v1:172.21.15.28:6789/0 conn(0x7f5198157480 legacy=0x7f519806d430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.28:6789/0 2024-09-13T16:47:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.533+0000 7f518fe00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f51980c6270 0x7f51980c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.533+0000 7f518fe00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f51980c6270 0x7f51980c6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:37316/0 (socket says 172.21.15.28:37316) 2024-09-13T16:47:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.533+0000 7f518fe00640 1 -- 172.21.15.28:0/1401871150 learned_addr learned my addr 172.21.15.28:0/1401871150 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.534+0000 7f518fe00640 1 -- 172.21.15.28:0/1401871150 >> v2:172.21.15.28:3300/0 conn(0x7f51981570a0 msgr2=0x7f5198070040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.534+0000 7f518fe00640 1 --2- 172.21.15.28:0/1401871150 >> v2:172.21.15.28:3300/0 conn(0x7f51981570a0 0x7f5198070040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.534+0000 7f518fe00640 1 -- 172.21.15.28:0/1401871150 >> v1:172.21.15.28:6789/0 conn(0x7f5198157480 legacy=0x7f519806d430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.534+0000 7f518fe00640 1 -- 172.21.15.28:0/1401871150 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519815d600 con 0x7f51980c6270 2024-09-13T16:47:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.536+0000 7f518fe00640 1 --2- 172.21.15.28:0/1401871150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51980c6270 0x7f51980c6ef0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f51980c4470 tx=0x7f518802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94eca3a2a425d8c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.536+0000 7f5197400640 1 -- 172.21.15.28:0/1401871150 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188032440 con 0x7f51980c6270 2024-09-13T16:47:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.536+0000 7f5197400640 1 -- 172.21.15.28:0/1401871150 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51880325a0 con 0x7f51980c6270 2024-09-13T16:47:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.536+0000 7f5197400640 1 -- 172.21.15.28:0/1401871150 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188032930 con 0x7f51980c6270 2024-09-13T16:47:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.537+0000 7f519ee00640 1 -- 172.21.15.28:0/1401871150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51980c6270 msgr2=0x7f51980c6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.537+0000 7f519ee00640 1 --2- 172.21.15.28:0/1401871150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51980c6270 0x7f51980c6ef0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f51980c4470 tx=0x7f518802a250 comp rx=0 tx=0).stop 2024-09-13T16:47:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.538+0000 7f519ee00640 1 -- 172.21.15.28:0/1401871150 shutdown_connections 2024-09-13T16:47:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.538+0000 7f519ee00640 1 --2- 172.21.15.28:0/1401871150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51980c6270 0x7f51980c6ef0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.538+0000 7f519ee00640 1 --2- 172.21.15.28:0/1401871150 >> v2:172.21.15.28:3300/0 conn(0x7f51981570a0 0x7f5198070040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:20.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.538+0000 7f519ee00640 1 -- 172.21.15.28:0/1401871150 >> 172.21.15.28:0/1401871150 conn(0x7f5198069e80 msgr2=0x7f519806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:20.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.539+0000 7f519ee00640 1 -- 172.21.15.28:0/1401871150 shutdown_connections 2024-09-13T16:47:20.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.539+0000 7f519ee00640 1 -- 172.21.15.28:0/1401871150 wait complete. 2024-09-13T16:47:20.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.541+0000 7f519ee00640 1 Processor -- start 2024-09-13T16:47:20.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.541+0000 7f519ee00640 1 -- start start 2024-09-13T16:47:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.542+0000 7f519ee00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51980c6270 0x7f51981528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.542+0000 7f519ee00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51981570a0 0x7f51981541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.542+0000 7f519ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5198157480 0x7f51981f5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.542+0000 7f519ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51981518f0 con 0x7f5198157480 2024-09-13T16:47:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.542+0000 7f519ee00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5198157940 con 0x7f51981570a0 2024-09-13T16:47:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.542+0000 7f519ee00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5198157c00 con 0x7f51980c6270 2024-09-13T16:47:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.543+0000 7f519ce00640 1 -- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5198157480 msgr2=0x7f51981f5fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:47:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.543+0000 7f5197e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51980c6270 0x7f51981528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.543+0000 7f518fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51981570a0 0x7f51981541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.543+0000 7f519ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5198157480 0x7f51981f5fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.543+0000 7f5197e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51980c6270 0x7f51981528c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:43094/0 (socket says 172.21.15.28:43094) 2024-09-13T16:47:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.543+0000 7f518fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51981570a0 0x7f51981541f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:37318/0 (socket says 172.21.15.28:37318) 2024-09-13T16:47:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.543+0000 7f5197e00640 1 -- 172.21.15.28:0/367700430 learned_addr learned my addr 172.21.15.28:0/367700430 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.543+0000 7f5197e00640 1 -- 172.21.15.28:0/367700430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51981570a0 msgr2=0x7f51981541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.543+0000 7f5197e00640 1 --2- 172.21.15.28:0/367700430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51981570a0 0x7f51981541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.543+0000 7f5197e00640 1 -- 172.21.15.28:0/367700430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5198157480 msgr2=0x7f51981f5fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:47:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.543+0000 7f5197e00640 1 --2- 172.21.15.28:0/367700430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5198157480 0x7f51981f5fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.543+0000 7f5197e00640 1 -- 172.21.15.28:0/367700430 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51880094d0 con 0x7f51980c6270 2024-09-13T16:47:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.543+0000 7f5197e00640 1 --2- 172.21.15.28:0/367700430 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51980c6270 0x7f51981528c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f518000b550 tx=0x7f518000ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.544+0000 7f5194c00640 1 -- 172.21.15.28:0/367700430 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5180003dc0 con 0x7f51980c6270 2024-09-13T16:47:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.544+0000 7f5194c00640 1 -- 172.21.15.28:0/367700430 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5180005070 con 0x7f51980c6270 2024-09-13T16:47:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.544+0000 7f5194c00640 1 -- 172.21.15.28:0/367700430 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5180005420 con 0x7f51980c6270 2024-09-13T16:47:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.545+0000 7f519ee00640 1 -- 172.21.15.28:0/367700430 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5198151b90 con 0x7f51980c6270 2024-09-13T16:47:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.545+0000 7f519ee00640 1 -- 172.21.15.28:0/367700430 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51980c9e60 con 0x7f51980c6270 2024-09-13T16:47:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.546+0000 7f519ee00640 1 -- 172.21.15.28:0/367700430 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51981520b0 con 0x7f51980c6270 2024-09-13T16:47:20.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.547+0000 7f5194c00640 1 -- 172.21.15.28:0/367700430 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5180005580 con 0x7f51980c6270 2024-09-13T16:47:20.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.548+0000 7f5194c00640 1 --2- 172.21.15.28:0/367700430 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f517406abb0 0x7f517406d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:20.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.548+0000 7f5194c00640 1 -- 172.21.15.28:0/367700430 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5180089380 con 0x7f51980c6270 2024-09-13T16:47:20.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.549+0000 7f518fe00640 1 --2- 172.21.15.28:0/367700430 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f517406abb0 0x7f517406d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.549+0000 7f518fe00640 1 --2- 172.21.15.28:0/367700430 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f517406abb0 0x7f517406d070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5188034000 tx=0x7f518802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:20.553+0000 7f5194c00640 1 -- 172.21.15.28:0/367700430 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f51981520b0 con 0x7f51980c6270 2024-09-13T16:47:21.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:21.136+0000 7f519ee00640 1 -- 172.21.15.28:0/367700430 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f51981520b0 con 0x7f51980c6270 2024-09-13T16:47:21.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:21.137+0000 7f5194c00640 1 -- 172.21.15.28:0/367700430 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f5180054440 con 0x7f51980c6270 2024-09-13T16:47:21.139 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:47:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:21.141+0000 7f519ee00640 1 -- 172.21.15.28:0/367700430 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f517406abb0 msgr2=0x7f517406d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:21.141+0000 7f519ee00640 1 --2- 172.21.15.28:0/367700430 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f517406abb0 0x7f517406d070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5188034000 tx=0x7f518802e000 comp rx=0 tx=0).stop 2024-09-13T16:47:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:21.142+0000 7f519ee00640 1 -- 172.21.15.28:0/367700430 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51980c6270 msgr2=0x7f51981528c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:21.142+0000 7f519ee00640 1 --2- 172.21.15.28:0/367700430 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51980c6270 0x7f51981528c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f518000b550 tx=0x7f518000ba10 comp rx=0 tx=0).stop 2024-09-13T16:47:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:21.142+0000 7f519ee00640 1 -- 172.21.15.28:0/367700430 shutdown_connections 2024-09-13T16:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:21.142+0000 7f519ee00640 1 --2- 172.21.15.28:0/367700430 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f517406abb0 0x7f517406d070 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:21.142+0000 7f519ee00640 1 --2- 172.21.15.28:0/367700430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5198157480 0x7f51981f5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:21.142+0000 7f519ee00640 1 --2- 172.21.15.28:0/367700430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51981570a0 0x7f51981541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:21.142+0000 7f519ee00640 1 --2- 172.21.15.28:0/367700430 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51980c6270 0x7f51981528c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:21.142+0000 7f519ee00640 1 -- 172.21.15.28:0/367700430 >> 172.21.15.28:0/367700430 conn(0x7f5198069e80 msgr2=0x7f519806dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:21.143+0000 7f519ee00640 1 -- 172.21.15.28:0/367700430 shutdown_connections 2024-09-13T16:47:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:21.143+0000 7f519ee00640 1 -- 172.21.15.28:0/367700430 wait complete. 2024-09-13T16:47:24.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:47:24.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.380+0000 7fc0b0400640 1 Processor -- start 2024-09-13T16:47:24.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.380+0000 7fc0b0400640 1 -- start start 2024-09-13T16:47:24.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.382+0000 7fc0b0400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0a8072170 0x7fc0a80c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:24.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.382+0000 7fc0b0400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc0a80c6350 0x7fc0a81559d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:24.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.382+0000 7fc0b0400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a80c42b0 con 0x7fc0a814fa80 2024-09-13T16:47:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.382+0000 7fc0b0400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0a80c36e0 con 0x7fc0a80c6350 2024-09-13T16:47:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.382+0000 7fc0b0400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a80c3980 con 0x7fc0a8072170 2024-09-13T16:47:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.382+0000 7fc0ada00640 1 -- >> v1:172.21.15.28:6789/0 conn(0x7fc0a814fa80 legacy=0x7fc0a80c6ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.28:6789/0 2024-09-13T16:47:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.382+0000 7fc0ae400640 1 -- >> v2:172.21.15.28:3300/0 conn(0x7fc0a80c6350 msgr2=0x7fc0a81559d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:47:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.382+0000 7fc0ae400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc0a80c6350 0x7fc0a81559d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.382+0000 7fc0ad000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0a8072170 0x7fc0a80c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.382+0000 7fc0ad000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0a8072170 0x7fc0a80c1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:43096/0 (socket says 172.21.15.28:43096) 2024-09-13T16:47:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.382+0000 7fc0ad000640 1 -- 172.21.15.28:0/1792420340 learned_addr learned my addr 172.21.15.28:0/1792420340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.383+0000 7fc0ad000640 1 -- 172.21.15.28:0/1792420340 >> v2:172.21.15.28:3300/0 conn(0x7fc0a80c6350 msgr2=0x7fc0a81559d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.383+0000 7fc0ad000640 1 --2- 172.21.15.28:0/1792420340 >> v2:172.21.15.28:3300/0 conn(0x7fc0a80c6350 0x7fc0a81559d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.383+0000 7fc0ad000640 1 -- 172.21.15.28:0/1792420340 >> v1:172.21.15.28:6789/0 conn(0x7fc0a814fa80 legacy=0x7fc0a80c6ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:24.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.383+0000 7fc0ad000640 1 -- 172.21.15.28:0/1792420340 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a80c3c60 con 0x7fc0a8072170 2024-09-13T16:47:24.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.384+0000 7fc0ad000640 1 --2- 172.21.15.28:0/1792420340 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0a8072170 0x7fc0a80c1960 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc094009870 tx=0x7fc09402a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c2e2308c147f2f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:24.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.384+0000 7fc0a7e00640 1 -- 172.21.15.28:0/1792420340 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094032440 con 0x7fc0a8072170 2024-09-13T16:47:24.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.384+0000 7fc0a7e00640 1 -- 172.21.15.28:0/1792420340 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0940325a0 con 0x7fc0a8072170 2024-09-13T16:47:24.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.384+0000 7fc0a7e00640 1 -- 172.21.15.28:0/1792420340 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094032930 con 0x7fc0a8072170 2024-09-13T16:47:24.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.385+0000 7fc0b0400640 1 -- 172.21.15.28:0/1792420340 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0a8072170 msgr2=0x7fc0a80c1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:24.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.385+0000 7fc0b0400640 1 --2- 172.21.15.28:0/1792420340 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0a8072170 0x7fc0a80c1960 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc094009870 tx=0x7fc09402a230 comp rx=0 tx=0).stop 2024-09-13T16:47:24.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.385+0000 7fc0b0400640 1 -- 172.21.15.28:0/1792420340 shutdown_connections 2024-09-13T16:47:24.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.385+0000 7fc0b0400640 1 --2- 172.21.15.28:0/1792420340 >> v2:172.21.15.28:3300/0 conn(0x7fc0a80c6350 0x7fc0a81559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:24.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.385+0000 7fc0b0400640 1 --2- 172.21.15.28:0/1792420340 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0a8072170 0x7fc0a80c1960 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:24.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.385+0000 7fc0b0400640 1 -- 172.21.15.28:0/1792420340 >> 172.21.15.28:0/1792420340 conn(0x7fc0a8069e80 msgr2=0x7fc0a806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:24.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.385+0000 7fc0b0400640 1 -- 172.21.15.28:0/1792420340 shutdown_connections 2024-09-13T16:47:24.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.385+0000 7fc0b0400640 1 -- 172.21.15.28:0/1792420340 wait complete. 2024-09-13T16:47:24.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.387+0000 7fc0b0400640 1 Processor -- start 2024-09-13T16:47:24.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.387+0000 7fc0b0400640 1 -- start start 2024-09-13T16:47:24.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.388+0000 7fc0b0400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0a8072170 0x7fc0a80b9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:24.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.388+0000 7fc0b0400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc0a80c6350 0x7fc0a80b9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:24.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.388+0000 7fc0b0400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0a814fa80 0x7fc0a80b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:24.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.388+0000 7fc0b0400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a80b8830 con 0x7fc0a814fa80 2024-09-13T16:47:24.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.388+0000 7fc0b0400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a81ee980 con 0x7fc0a80c6350 2024-09-13T16:47:24.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.388+0000 7fc0b0400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a81ee6e0 con 0x7fc0a8072170 2024-09-13T16:47:24.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.388+0000 7fc0ae400640 1 -- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0a814fa80 msgr2=0x7fc0a80b7170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:47:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.388+0000 7fc0ae400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0a814fa80 0x7fc0a80b7170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.388+0000 7fc0ada00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0a8072170 0x7fc0a80b9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.388+0000 7fc0ada00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0a8072170 0x7fc0a80b9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:43112/0 (socket says 172.21.15.28:43112) 2024-09-13T16:47:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.388+0000 7fc0ada00640 1 -- 172.21.15.28:0/4254163116 learned_addr learned my addr 172.21.15.28:0/4254163116 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.388+0000 7fc0ad000640 1 --2- 172.21.15.28:0/4254163116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc0a80c6350 0x7fc0a80b9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.389+0000 7fc0ada00640 1 -- 172.21.15.28:0/4254163116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc0a80c6350 msgr2=0x7fc0a80b9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.389+0000 7fc0ada00640 1 --2- 172.21.15.28:0/4254163116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc0a80c6350 0x7fc0a80b9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.389+0000 7fc0ada00640 1 -- 172.21.15.28:0/4254163116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0a814fa80 msgr2=0x7fc0a80b7170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:47:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.389+0000 7fc0ada00640 1 --2- 172.21.15.28:0/4254163116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0a814fa80 0x7fc0a80b7170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.389+0000 7fc0ada00640 1 -- 172.21.15.28:0/4254163116 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0940094d0 con 0x7fc0a8072170 2024-09-13T16:47:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.389+0000 7fc0ada00640 1 --2- 172.21.15.28:0/4254163116 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0a8072170 0x7fc0a80b9aa0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc09c00b550 tx=0x7fc09c00ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.389+0000 7fc0a5600640 1 -- 172.21.15.28:0/4254163116 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c003dc0 con 0x7fc0a8072170 2024-09-13T16:47:24.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.389+0000 7fc0a5600640 1 -- 172.21.15.28:0/4254163116 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09c005070 con 0x7fc0a8072170 2024-09-13T16:47:24.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.389+0000 7fc0a5600640 1 -- 172.21.15.28:0/4254163116 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c005420 con 0x7fc0a8072170 2024-09-13T16:47:24.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.389+0000 7fc0b0400640 1 -- 172.21.15.28:0/4254163116 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a80b8fc0 con 0x7fc0a8072170 2024-09-13T16:47:24.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.390+0000 7fc0b0400640 1 -- 172.21.15.28:0/4254163116 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a80b9580 con 0x7fc0a8072170 2024-09-13T16:47:24.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.391+0000 7fc0b0400640 1 -- 172.21.15.28:0/4254163116 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a814cda0 con 0x7fc0a8072170 2024-09-13T16:47:24.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.392+0000 7fc0a5600640 1 -- 172.21.15.28:0/4254163116 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc09c005580 con 0x7fc0a8072170 2024-09-13T16:47:24.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.393+0000 7fc0a5600640 1 --2- 172.21.15.28:0/4254163116 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc08806abb0 0x7fc08806d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:24.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.393+0000 7fc0a5600640 1 -- 172.21.15.28:0/4254163116 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc09c089300 con 0x7fc0a8072170 2024-09-13T16:47:24.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.393+0000 7fc0ad000640 1 --2- 172.21.15.28:0/4254163116 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc08806abb0 0x7fc08806d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:24.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.394+0000 7fc0ad000640 1 --2- 172.21.15.28:0/4254163116 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc08806abb0 0x7fc08806d070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc094034000 tx=0x7fc09402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:24.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.397+0000 7fc0a5600640 1 -- 172.21.15.28:0/4254163116 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc09c054440 con 0x7fc0a8072170 2024-09-13T16:47:24.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.868+0000 7fc0b0400640 1 -- 172.21.15.28:0/4254163116 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc0a8153920 con 0x7fc0a8072170 2024-09-13T16:47:24.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.869+0000 7fc0a5600640 1 -- 172.21.15.28:0/4254163116 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7fc09c057a60 con 0x7fc0a8072170 2024-09-13T16:47:24.870 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:47:24.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.872+0000 7fc0b0400640 1 -- 172.21.15.28:0/4254163116 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc08806abb0 msgr2=0x7fc08806d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:24.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.872+0000 7fc0b0400640 1 --2- 172.21.15.28:0/4254163116 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc08806abb0 0x7fc08806d070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc094034000 tx=0x7fc09402e000 comp rx=0 tx=0).stop 2024-09-13T16:47:24.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.872+0000 7fc0b0400640 1 -- 172.21.15.28:0/4254163116 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0a8072170 msgr2=0x7fc0a80b9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:24.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.872+0000 7fc0b0400640 1 --2- 172.21.15.28:0/4254163116 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0a8072170 0x7fc0a80b9aa0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc09c00b550 tx=0x7fc09c00ba10 comp rx=0 tx=0).stop 2024-09-13T16:47:24.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.872+0000 7fc0b0400640 1 -- 172.21.15.28:0/4254163116 shutdown_connections 2024-09-13T16:47:24.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.872+0000 7fc0b0400640 1 --2- 172.21.15.28:0/4254163116 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc08806abb0 0x7fc08806d070 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:24.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.872+0000 7fc0b0400640 1 --2- 172.21.15.28:0/4254163116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0a814fa80 0x7fc0a80b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:24.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.872+0000 7fc0b0400640 1 --2- 172.21.15.28:0/4254163116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc0a80c6350 0x7fc0a80b9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:24.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.872+0000 7fc0b0400640 1 --2- 172.21.15.28:0/4254163116 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0a8072170 0x7fc0a80b9aa0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:24.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.872+0000 7fc0b0400640 1 -- 172.21.15.28:0/4254163116 >> 172.21.15.28:0/4254163116 conn(0x7fc0a8069e80 msgr2=0x7fc0a806d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:24.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.872+0000 7fc0b0400640 1 -- 172.21.15.28:0/4254163116 shutdown_connections 2024-09-13T16:47:24.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:24.873+0000 7fc0b0400640 1 -- 172.21.15.28:0/4254163116 wait complete. 2024-09-13T16:47:27.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:47:28.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.145+0000 7faa69400640 1 Processor -- start 2024-09-13T16:47:28.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.146+0000 7faa69400640 1 -- start start 2024-09-13T16:47:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.149+0000 7faa69400640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7faa641570a0 0x7faa64070040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.149+0000 7faa69400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa640c6270 0x7faa640c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.149+0000 7faa69400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6415df30 con 0x7faa64157480 2024-09-13T16:47:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.149+0000 7faa69400640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa64161340 con 0x7faa641570a0 2024-09-13T16:47:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.149+0000 7faa69400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa6415d320 con 0x7faa640c6270 2024-09-13T16:47:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.149+0000 7faa63200640 1 -- >> v1:172.21.15.28:6789/0 conn(0x7faa64157480 legacy=0x7faa6406d430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.28:6789/0 2024-09-13T16:47:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.150+0000 7faa62800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7faa641570a0 0x7faa64070040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.150+0000 7faa62800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7faa641570a0 0x7faa64070040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:37326/0 (socket says 172.21.15.28:37326) 2024-09-13T16:47:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.150+0000 7faa62800640 1 -- 172.21.15.28:0/1759162820 learned_addr learned my addr 172.21.15.28:0/1759162820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.150+0000 7faa61e00640 1 --2- 172.21.15.28:0/1759162820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa640c6270 0x7faa640c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.150+0000 7faa62800640 1 -- 172.21.15.28:0/1759162820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa640c6270 msgr2=0x7faa640c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:47:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.150+0000 7faa62800640 1 --2- 172.21.15.28:0/1759162820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa640c6270 0x7faa640c6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.150+0000 7faa62800640 1 -- 172.21.15.28:0/1759162820 >> v1:172.21.15.28:6789/0 conn(0x7faa64157480 legacy=0x7faa6406d430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.150+0000 7faa62800640 1 -- 172.21.15.28:0/1759162820 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6415d600 con 0x7faa641570a0 2024-09-13T16:47:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.150+0000 7faa61e00640 1 --2- 172.21.15.28:0/1759162820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa640c6270 0x7faa640c6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:47:28.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.155+0000 7faa62800640 1 --2- 172.21.15.28:0/1759162820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa641570a0 0x7faa64070040 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faa5c009e50 tx=0x7faa5c02a6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4da9ebcfe6caf063 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:28.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.156+0000 7faa61400640 1 -- 172.21.15.28:0/1759162820 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c02aea0 con 0x7faa641570a0 2024-09-13T16:47:28.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.156+0000 7faa61400640 1 -- 172.21.15.28:0/1759162820 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5c02d050 con 0x7faa641570a0 2024-09-13T16:47:28.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.156+0000 7faa61400640 1 -- 172.21.15.28:0/1759162820 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c031820 con 0x7faa641570a0 2024-09-13T16:47:28.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.157+0000 7faa69400640 1 -- 172.21.15.28:0/1759162820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa641570a0 msgr2=0x7faa64070040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:28.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.157+0000 7faa69400640 1 --2- 172.21.15.28:0/1759162820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa641570a0 0x7faa64070040 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faa5c009e50 tx=0x7faa5c02a6c0 comp rx=0 tx=0).stop 2024-09-13T16:47:28.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.157+0000 7faa69400640 1 -- 172.21.15.28:0/1759162820 shutdown_connections 2024-09-13T16:47:28.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.157+0000 7faa69400640 1 --2- 172.21.15.28:0/1759162820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa640c6270 0x7faa640c6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:28.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.157+0000 7faa69400640 1 --2- 172.21.15.28:0/1759162820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa641570a0 0x7faa64070040 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:28.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.157+0000 7faa69400640 1 -- 172.21.15.28:0/1759162820 >> 172.21.15.28:0/1759162820 conn(0x7faa64069e80 msgr2=0x7faa6406a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:28.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.157+0000 7faa69400640 1 -- 172.21.15.28:0/1759162820 shutdown_connections 2024-09-13T16:47:28.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.157+0000 7faa69400640 1 -- 172.21.15.28:0/1759162820 wait complete. 2024-09-13T16:47:28.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.161+0000 7faa69400640 1 Processor -- start 2024-09-13T16:47:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.162+0000 7faa69400640 1 -- start start 2024-09-13T16:47:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.163+0000 7faa69400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa640c6270 0x7faa64156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.163+0000 7faa69400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa641570a0 0x7faa64157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.163+0000 7faa69400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa64157480 0x7faa64157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.163+0000 7faa69400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa640b8570 con 0x7faa641570a0 2024-09-13T16:47:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.163+0000 7faa69400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa64156d90 con 0x7faa640c6270 2024-09-13T16:47:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.163+0000 7faa69400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa640b8130 con 0x7faa64157480 2024-09-13T16:47:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.163+0000 7faa61e00640 1 -- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa641570a0 msgr2=0x7faa64157940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:47:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.163+0000 7faa61e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa641570a0 0x7faa64157940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.163+0000 7faa62800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa640c6270 0x7faa64156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.164+0000 7faa62800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa640c6270 0x7faa64156850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:37340/0 (socket says 172.21.15.28:37340) 2024-09-13T16:47:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.164+0000 7faa62800640 1 -- 172.21.15.28:0/2336386399 learned_addr learned my addr 172.21.15.28:0/2336386399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.164+0000 7faa63200640 1 --2- 172.21.15.28:0/2336386399 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa64157480 0x7faa64157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.164+0000 7faa62800640 1 -- 172.21.15.28:0/2336386399 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa64157480 msgr2=0x7faa64157e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.164+0000 7faa62800640 1 --2- 172.21.15.28:0/2336386399 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa64157480 0x7faa64157e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.164+0000 7faa62800640 1 -- 172.21.15.28:0/2336386399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa641570a0 msgr2=0x7faa64157940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:47:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.164+0000 7faa62800640 1 --2- 172.21.15.28:0/2336386399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa641570a0 0x7faa64157940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.164+0000 7faa62800640 1 -- 172.21.15.28:0/2336386399 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa54009400 con 0x7faa640c6270 2024-09-13T16:47:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.164+0000 7faa62800640 1 --2- 172.21.15.28:0/2336386399 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa640c6270 0x7faa64156850 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faa5c009e50 tx=0x7faa5c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.165+0000 7faa5aa00640 1 -- 172.21.15.28:0/2336386399 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c03d860 con 0x7faa640c6270 2024-09-13T16:47:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.165+0000 7faa5aa00640 1 -- 172.21.15.28:0/2336386399 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5c03d9c0 con 0x7faa640c6270 2024-09-13T16:47:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.165+0000 7faa5aa00640 1 -- 172.21.15.28:0/2336386399 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c03ddb0 con 0x7faa640c6270 2024-09-13T16:47:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.165+0000 7faa69400640 1 -- 172.21.15.28:0/2336386399 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa5c009b30 con 0x7faa640c6270 2024-09-13T16:47:28.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.165+0000 7faa69400640 1 -- 172.21.15.28:0/2336386399 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa64159e00 con 0x7faa640c6270 2024-09-13T16:47:28.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.166+0000 7faa69400640 1 -- 172.21.15.28:0/2336386399 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa64070040 con 0x7faa640c6270 2024-09-13T16:47:28.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.168+0000 7faa5aa00640 1 -- 172.21.15.28:0/2336386399 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faa5c003990 con 0x7faa640c6270 2024-09-13T16:47:28.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.169+0000 7faa5aa00640 1 --2- 172.21.15.28:0/2336386399 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7faa4406abb0 0x7faa4406d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:28.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.170+0000 7faa5aa00640 1 -- 172.21.15.28:0/2336386399 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7faa5c0aa180 con 0x7faa640c6270 2024-09-13T16:47:28.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.170+0000 7faa61e00640 1 --2- 172.21.15.28:0/2336386399 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7faa4406abb0 0x7faa4406d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:28.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.171+0000 7faa61e00640 1 --2- 172.21.15.28:0/2336386399 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7faa4406abb0 0x7faa4406d070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faa641f3aa0 tx=0x7faa54009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:28.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.172+0000 7faa5aa00640 1 -- 172.21.15.28:0/2336386399 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7faa5c0752c0 con 0x7faa640c6270 2024-09-13T16:47:28.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.635+0000 7faa69400640 1 -- 172.21.15.28:0/2336386399 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7faa640c6890 con 0x7faa640c6270 2024-09-13T16:47:28.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.638+0000 7faa5aa00640 1 -- 172.21.15.28:0/2336386399 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7faa5c009510 con 0x7faa640c6270 2024-09-13T16:47:28.652 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:47:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.655+0000 7faa69400640 1 -- 172.21.15.28:0/2336386399 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7faa4406abb0 msgr2=0x7faa4406d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.655+0000 7faa69400640 1 --2- 172.21.15.28:0/2336386399 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7faa4406abb0 0x7faa4406d070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faa641f3aa0 tx=0x7faa54009190 comp rx=0 tx=0).stop 2024-09-13T16:47:28.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.655+0000 7faa69400640 1 -- 172.21.15.28:0/2336386399 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa640c6270 msgr2=0x7faa64156850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:28.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.655+0000 7faa69400640 1 --2- 172.21.15.28:0/2336386399 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa640c6270 0x7faa64156850 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faa5c009e50 tx=0x7faa5c004a40 comp rx=0 tx=0).stop 2024-09-13T16:47:28.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.655+0000 7faa69400640 1 -- 172.21.15.28:0/2336386399 shutdown_connections 2024-09-13T16:47:28.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.655+0000 7faa69400640 1 --2- 172.21.15.28:0/2336386399 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7faa4406abb0 0x7faa4406d070 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:28.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.655+0000 7faa69400640 1 --2- 172.21.15.28:0/2336386399 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa64157480 0x7faa64157e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:28.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.655+0000 7faa69400640 1 --2- 172.21.15.28:0/2336386399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa641570a0 0x7faa64157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:28.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.655+0000 7faa69400640 1 --2- 172.21.15.28:0/2336386399 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa640c6270 0x7faa64156850 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:28.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.655+0000 7faa69400640 1 -- 172.21.15.28:0/2336386399 >> 172.21.15.28:0/2336386399 conn(0x7faa64069e80 msgr2=0x7faa6415bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:28.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.659+0000 7faa69400640 1 -- 172.21.15.28:0/2336386399 shutdown_connections 2024-09-13T16:47:28.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:28.659+0000 7faa69400640 1 -- 172.21.15.28:0/2336386399 wait complete. 2024-09-13T16:47:31.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:47:31.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:31.917+0000 7f4ef8e00640 1 Processor -- start 2024-09-13T16:47:31.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:31.917+0000 7f4ef8e00640 1 -- start start 2024-09-13T16:47:31.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:31.919+0000 7f4ef8e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ef40cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:31.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:31.919+0000 7f4ef8e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ef4158f50 0x7f4ef40cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:31.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:31.919+0000 7f4ef8e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef40ca010 con 0x7f4ef40ca8b0 2024-09-13T16:47:31.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:31.919+0000 7f4ef8e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef40c9d70 con 0x7f4ef4158f50 2024-09-13T16:47:31.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:31.919+0000 7f4ef8e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40c9170 con 0x7f4ef4152cf0 2024-09-13T16:47:31.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:31.920+0000 7f4ef3400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ef40cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:31.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:31.920+0000 7f4ef3e00640 1 -- >> v2:172.21.15.28:3300/0 conn(0x7f4ef4158f50 msgr2=0x7f4ef40cb2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:47:31.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:31.920+0000 7f4ef3400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ef40cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:33928/0 (socket says 172.21.15.28:33928) 2024-09-13T16:47:31.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:31.920+0000 7f4ef3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ef4158f50 0x7f4ef40cb2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:31.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:31.920+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 learned_addr learned my addr 172.21.15.28:0/2348478805 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:31.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:31.920+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ef40cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:47:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:31.920+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ef40cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:47:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:31.920+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ef40cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:47:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:31.920+0000 7f4ef2a00640 1 -- >> v1:172.21.15.28:6789/0 conn(0x7f4ef40ca8b0 legacy=0x7f4ef4158840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.28:6789/0 2024-09-13T16:47:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:31.920+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ef40cad60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:32.121+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.28:3300/0 conn(0x7f4ef4158f50 msgr2=0x7f4ef40cb2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:47:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:32.121+0000 7f4ef2a00640 1 -- 172.21.15.28:0/2348478805 >> v1:172.21.15.28:6789/0 conn(0x7f4ef40ca8b0 legacy=0x7f4ef4158840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.28:6789/0 2024-09-13T16:47:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:32.121+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.28:3300/0 conn(0x7f4ef4158f50 0x7f4ef40cb2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T16:47:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:32.121+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ef40cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:32.121+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ef40cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T16:47:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:32.121+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ef40cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:47:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:32.121+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ef40cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:47:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:32.121+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ef40cad60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T16:47:32.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:32.521+0000 7f4ef2a00640 1 -- 172.21.15.28:0/2348478805 >> v1:172.21.15.28:6789/0 conn(0x7f4ef40ca8b0 legacy=0x7f4ef4158840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.28:6789/0 2024-09-13T16:47:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:32.522+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ef40cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:32.522+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ef40cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:47:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:32.522+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ef40cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:47:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:32.522+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ef40cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:47:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:32.522+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.28:3300/0 conn(0x7f4ef4158f50 msgr2=0x7f4ef40cb2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:47:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:32.522+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ef40cad60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T16:47:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:32.522+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.28:3300/0 conn(0x7f4ef4158f50 0x7f4ef40cb2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T16:47:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:33.322+0000 7f4ef2a00640 1 -- 172.21.15.28:0/2348478805 >> v1:172.21.15.28:6789/0 conn(0x7f4ef40ca8b0 legacy=0x7f4ef4158840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.28:6789/0 2024-09-13T16:47:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:33.323+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.28:3300/0 conn(0x7f4ef4158f50 msgr2=0x7f4ef40cb2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:47:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:33.323+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.28:3300/0 conn(0x7f4ef4158f50 0x7f4ef40cb2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T16:47:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:33.323+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ef40cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:33.323+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ef40cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:47:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:33.323+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ef40cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:47:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:33.323+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ef40cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:47:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:33.323+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ef40cad60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T16:47:34.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:47:34.386 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:47:34.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:34.919+0000 7f4ef0c00640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ef40cad60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:34.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:34.919+0000 7f4ef0c00640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ef40cad60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:34.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:34.919+0000 7f4ef0c00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.28:3300/0 conn(0x7f4ef4158f50 msgr2=0x7f4ef40cb2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:34.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:34.919+0000 7f4ef0c00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.28:3300/0 conn(0x7f4ef4158f50 0x7f4ef40cb2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:34.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:34.919+0000 7f4ef0c00640 1 -- 172.21.15.28:0/2348478805 >> v1:172.21.15.28:6789/0 conn(0x7f4ef40ca8b0 legacy=0x7f4ef4158840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:34.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:34.919+0000 7f4ef0c00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:34.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:34.919+0000 7f4ef0c00640 1 -- 172.21.15.28:0/2348478805 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef415cb80 con 0x7f4ed40036c0 2024-09-13T16:47:34.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:34.919+0000 7f4ef0c00640 1 -- 172.21.15.28:0/2348478805 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef40c9d70 con 0x7f4ed4007370 2024-09-13T16:47:34.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:34.920+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:34.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:34.920+0000 7f4ef2a00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:47:34.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:34.920+0000 7f4ef2a00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:47:34.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:34.920+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:47:34.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:34.920+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:35.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:35.121+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:35.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:35.121+0000 7f4ef2a00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T16:47:35.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:35.121+0000 7f4ef2a00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:47:35.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:35.121+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:47:35.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:35.121+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T16:47:35.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:35.522+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:35.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:35.522+0000 7f4ef2a00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T16:47:35.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:35.522+0000 7f4ef2a00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:47:35.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:35.522+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:47:35.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:35.522+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T16:47:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:36.325+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:36.325+0000 7f4ef2a00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T16:47:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:36.325+0000 7f4ef2a00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:47:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:36.325+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:47:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:36.325+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T16:47:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.920+0000 7f4ef0c00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:37.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.920+0000 7f4ef0c00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:37.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.920+0000 7f4ef0c00640 1 -- 172.21.15.28:0/2348478805 >> v1:172.21.15.191:6789/0 conn(0x7f4ed40036c0 legacy=0x7f4ed4005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:37.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.920+0000 7f4ef0c00640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ed4009f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:37.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.920+0000 7f4ef0c00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:37.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.920+0000 7f4ef0c00640 1 -- 172.21.15.28:0/2348478805 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef40ca010 con 0x7f4ed400a4a0 2024-09-13T16:47:37.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.920+0000 7f4ef0c00640 1 -- 172.21.15.28:0/2348478805 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef40c9d70 con 0x7f4ed4007370 2024-09-13T16:47:37.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.920+0000 7f4ef0c00640 1 -- 172.21.15.28:0/2348478805 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40c9170 con 0x7f4ef4152cf0 2024-09-13T16:47:37.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.920+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2348478805 reap_dead start 2024-09-13T16:47:37.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.920+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ed4009f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:37.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.920+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.920+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ed4009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:47:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.920+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ed4009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:47:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.920+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ed4009f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:47:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.921+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ed4009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.921+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T16:47:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.921+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:47:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.921+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:47:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:37.921+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:38.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.123+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ed4009f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:38.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.124+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ed4009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T16:47:38.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.124+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ed4009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:47:38.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.124+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ed4009f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:47:38.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.124+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ed4009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T16:47:38.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.125+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:38.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.125+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:47:38.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.125+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:47:38.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.125+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:47:38.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.125+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T16:47:38.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.526+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:38.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.527+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:47:38.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.527+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ed4009f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:38.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.527+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:47:38.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.527+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:47:38.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.527+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T16:47:38.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.528+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ed4009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T16:47:38.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.528+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ed4009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:47:38.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.528+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ed4009f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:47:38.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:38.528+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ed4009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T16:47:39.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:39.328+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:39.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:39.328+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:47:39.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:39.328+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:47:39.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:39.328+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:47:39.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:39.328+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T16:47:39.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:39.329+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ed4009f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:39.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:39.330+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ed4009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T16:47:39.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:39.330+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ed4009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:47:39.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:39.330+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ed4009f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:47:39.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:39.330+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ed4009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T16:47:40.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.094+0000 7f4ef2000640 1 -- 172.21.15.28:0/2348478805 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030806987 0 0) 0x7f4ef40ca010 con 0x7f4ed400a4a0 2024-09-13T16:47:40.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.094+0000 7f4ef2000640 1 -- 172.21.15.28:0/2348478805 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ed8003520 con 0x7f4ed400a4a0 2024-09-13T16:47:40.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.098+0000 7f4ef2000640 1 -- 172.21.15.28:0/2348478805 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4140735733 0 0) 0x7f4ed8003520 con 0x7f4ed400a4a0 2024-09-13T16:47:40.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.099+0000 7f4ef2000640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ed4009f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:40.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.099+0000 7f4ef2000640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ed4009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.099+0000 7f4ef2000640 1 -- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 msgr2=0x7f4ed4009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:40.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.099+0000 7f4ef2000640 1 --2- 172.21.15.28:0/2348478805 >> v2:172.21.15.191:3300/0 conn(0x7f4ed4007370 0x7f4ed4009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.099+0000 7f4ef2000640 1 -- 172.21.15.28:0/2348478805 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed400ab00 con 0x7f4ed400a4a0 2024-09-13T16:47:40.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.099+0000 7f4ef2000640 1 -- 172.21.15.28:0/2348478805 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1654382301 0 0) 0x7f4ee0009860 con 0x7f4ed400a4a0 2024-09-13T16:47:40.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.099+0000 7f4ef2000640 1 -- 172.21.15.28:0/2348478805 >> v1:172.21.15.191:6789/0 conn(0x7f4ed400a4a0 legacy=0x7f4ed400f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.099+0000 7f4ef2000640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed8003970 0x7f4ed8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.099+0000 7f4ef2000640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ed80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.099+0000 7f4ef2000640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ed8004d00 0x7f4ed8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.099+0000 7f4ef2000640 1 -- 172.21.15.28:0/2348478805 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40c9d70 con 0x7f4ed8003970 2024-09-13T16:47:40.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.099+0000 7f4ef2000640 1 -- 172.21.15.28:0/2348478805 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40c9170 con 0x7f4ed8004d00 2024-09-13T16:47:40.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.099+0000 7f4ef2000640 1 -- 172.21.15.28:0/2348478805 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed800ba80 con 0x7f4ef4152cf0 2024-09-13T16:47:40.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.101+0000 7f4ef3400640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed8003970 msgr2=0x7f4ed8003d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:47:40.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.101+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ed8004d00 0x7f4ed8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.101+0000 7f4ef3400640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed8003970 0x7f4ed8003d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:40.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.101+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ed80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.102+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 msgr2=0x7f4ed80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:47:40.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.102+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef4152cf0 0x7f4ed80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.102+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed8003970 msgr2=0x7f4ed8003d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:40.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.102+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed8003970 0x7f4ed8003d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.102+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2348478805 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed400ab00 con 0x7f4ed8004d00 2024-09-13T16:47:40.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.105+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ed8004d00 0x7f4ed8005180 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4ef40c3a20 tx=0x7f4ed80057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64f330b2a0391ae4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.106+0000 7f4ef2000640 1 -- 172.21.15.28:0/2348478805 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec030480 con 0x7f4ed8004d00 2024-09-13T16:47:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.107+0000 7f4ef2000640 1 -- 172.21.15.28:0/2348478805 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eec030600 con 0x7f4ed8004d00 2024-09-13T16:47:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.107+0000 7f4ef2000640 1 -- 172.21.15.28:0/2348478805 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec0309b0 con 0x7f4ed8004d00 2024-09-13T16:47:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.108+0000 7f4ef8e00640 1 -- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ed8004d00 msgr2=0x7f4ed8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.108+0000 7f4ef8e00640 1 --2- 172.21.15.28:0/2348478805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ed8004d00 0x7f4ed8005180 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4ef40c3a20 tx=0x7f4ed80057f0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.109+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2348478805 reap_dead start 2024-09-13T16:47:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.111+0000 7f4ef8e00640 1 -- 172.21.15.28:0/2348478805 shutdown_connections 2024-09-13T16:47:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.111+0000 7f4ef8e00640 1 -- 172.21.15.28:0/2348478805 >> 172.21.15.28:0/2348478805 conn(0x7f4ef4065c50 msgr2=0x7f4ef4066060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:40.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.111+0000 7f4ef8e00640 1 -- 172.21.15.28:0/2348478805 shutdown_connections 2024-09-13T16:47:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.111+0000 7f4ef8e00640 1 -- 172.21.15.28:0/2348478805 wait complete. 2024-09-13T16:47:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.114+0000 7f4ef8e00640 1 Processor -- start 2024-09-13T16:47:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.115+0000 7f4ef8e00640 1 -- start start 2024-09-13T16:47:40.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.117+0000 7f4ef8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed8004d00 0x7f4ef415bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.117+0000 7f4ef8e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef40c7860 0x7f4ef41ea870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.117+0000 7f4ef3400640 1 -- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed8004d00 msgr2=0x7f4ef415bb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:47:40.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.118+0000 7f4ef8e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ef41eadb0 0x7f4ef4148b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.118+0000 7f4ef3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed8004d00 0x7f4ef415bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.118+0000 7f4ef2a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef40c7860 0x7f4ef41ea870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.118+0000 7f4ef8e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef4156050 con 0x7f4ed8004d00 2024-09-13T16:47:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.118+0000 7f4ef8e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef41eb270 con 0x7f4ef41eadb0 2024-09-13T16:47:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.118+0000 7f4ef8e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef415b570 con 0x7f4ef40c7860 2024-09-13T16:47:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.118+0000 7f4ef3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ef41eadb0 0x7f4ef4148b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.119+0000 7f4ef3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ef41eadb0 0x7f4ef4148b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:38948/0 (socket says 172.21.15.28:38948) 2024-09-13T16:47:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.119+0000 7f4ef3e00640 1 -- 172.21.15.28:0/1307862567 learned_addr learned my addr 172.21.15.28:0/1307862567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.120+0000 7f4ef3e00640 1 -- 172.21.15.28:0/1307862567 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef40c7860 msgr2=0x7f4ef41ea870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.120+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/1307862567 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef40c7860 0x7f4ef41ea870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.120+0000 7f4ef3e00640 1 -- 172.21.15.28:0/1307862567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed8004d00 msgr2=0x7f4ef415bb30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:47:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.120+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/1307862567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed8004d00 0x7f4ef415bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.120+0000 7f4ef3e00640 1 -- 172.21.15.28:0/1307862567 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef415b850 con 0x7f4ef41eadb0 2024-09-13T16:47:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.121+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/1307862567 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ef41eadb0 0x7f4ef4148b00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4eec02c060 tx=0x7f4eec006080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.121+0000 7f4eeb400640 1 -- 172.21.15.28:0/1307862567 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec047070 con 0x7f4ef41eadb0 2024-09-13T16:47:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.121+0000 7f4eeb400640 1 -- 172.21.15.28:0/1307862567 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eec03f040 con 0x7f4ef41eadb0 2024-09-13T16:47:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.121+0000 7f4eeb400640 1 -- 172.21.15.28:0/1307862567 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec030480 con 0x7f4ef41eadb0 2024-09-13T16:47:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.121+0000 7f4ef8e00640 1 -- 172.21.15.28:0/1307862567 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef40c7060 con 0x7f4ef41eadb0 2024-09-13T16:47:40.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.122+0000 7f4ef8e00640 1 -- 172.21.15.28:0/1307862567 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef415c160 con 0x7f4ef41eadb0 2024-09-13T16:47:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.123+0000 7f4ef8e00640 1 -- 172.21.15.28:0/1307862567 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef40ca720 con 0x7f4ef41eadb0 2024-09-13T16:47:40.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.126+0000 7f4eeb400640 1 -- 172.21.15.28:0/1307862567 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4eec039660 con 0x7f4ef41eadb0 2024-09-13T16:47:40.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.128+0000 7f4eeb400640 1 --2- 172.21.15.28:0/1307862567 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4ed406bcb0 0x7f4ed406e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.128+0000 7f4eeb400640 1 -- 172.21.15.28:0/1307862567 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4eec0aaf70 con 0x7f4ef41eadb0 2024-09-13T16:47:40.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.128+0000 7f4ef3400640 1 --2- 172.21.15.28:0/1307862567 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4ed406bcb0 0x7f4ed406e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.129+0000 7f4ef3400640 1 --2- 172.21.15.28:0/1307862567 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4ed406bcb0 0x7f4ed406e170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4006e00 tx=0x7f4ee4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:40.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.130+0000 7f4eeb400640 1 -- 172.21.15.28:0/1307862567 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4eec079bd0 con 0x7f4ef41eadb0 2024-09-13T16:47:40.537 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:47:40.537 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 rewriters = 365395.56 kB/sec 2024-09-13T16:47:40.537 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 rewriters = 365385.19 kB/sec 2024-09-13T16:47:40.538 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 365395.56 kB/sec 2024-09-13T16:47:40.538 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 365395.56 kB/sec 2024-09-13T16:47:40.538 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 365395.56 kB/sec 2024-09-13T16:47:40.538 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 10485760.00 kB 2024-09-13T16:47:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.563+0000 7f4ef8e00640 1 -- 172.21.15.28:0/1307862567 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4ef40c6ec0 con 0x7f4ef41eadb0 2024-09-13T16:47:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.563+0000 7f4eeb400640 1 -- 172.21.15.28:0/1307862567 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f4ef40c6ec0 con 0x7f4ef41eadb0 2024-09-13T16:47:40.565 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":26,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:47:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.566+0000 7f4ef8e00640 1 -- 172.21.15.28:0/1307862567 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4ed406bcb0 msgr2=0x7f4ed406e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.566+0000 7f4ef8e00640 1 --2- 172.21.15.28:0/1307862567 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4ed406bcb0 0x7f4ed406e170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4006e00 tx=0x7f4ee4008040 comp rx=0 tx=0).stop 2024-09-13T16:47:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.566+0000 7f4ef8e00640 1 -- 172.21.15.28:0/1307862567 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ef41eadb0 msgr2=0x7f4ef4148b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.566+0000 7f4ef8e00640 1 --2- 172.21.15.28:0/1307862567 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ef41eadb0 0x7f4ef4148b00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4eec02c060 tx=0x7f4eec006080 comp rx=0 tx=0).stop 2024-09-13T16:47:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.566+0000 7f4ef8e00640 1 -- 172.21.15.28:0/1307862567 shutdown_connections 2024-09-13T16:47:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.566+0000 7f4ef8e00640 1 --2- 172.21.15.28:0/1307862567 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4ed406bcb0 0x7f4ed406e170 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.566+0000 7f4ef8e00640 1 --2- 172.21.15.28:0/1307862567 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ef41eadb0 0x7f4ef4148b00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.566+0000 7f4ef8e00640 1 --2- 172.21.15.28:0/1307862567 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ef40c7860 0x7f4ef41ea870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.566+0000 7f4ef8e00640 1 --2- 172.21.15.28:0/1307862567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed8004d00 0x7f4ef415bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.566+0000 7f4ef8e00640 1 -- 172.21.15.28:0/1307862567 >> 172.21.15.28:0/1307862567 conn(0x7f4ef4065c50 msgr2=0x7f4ef40cad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:40.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.566+0000 7f4ef8e00640 1 -- 172.21.15.28:0/1307862567 shutdown_connections 2024-09-13T16:47:40.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.566+0000 7f4ef8e00640 1 -- 172.21.15.28:0/1307862567 wait complete. 2024-09-13T16:47:40.584 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-13T16:47:40.584 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-13T16:47:40.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T16:47:40.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.740+0000 7f98cbe00640 1 Processor -- start 2024-09-13T16:47:40.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.740+0000 7f98cbe00640 1 -- start start 2024-09-13T16:47:40.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.741+0000 7f98cbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98cc14f880 0x7f98cc155d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.741+0000 7f98cbe00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f98cc1562d0 0x7f98cc15ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.741+0000 7f98cbe00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98cc0c6350 con 0x7f98cc06fd80 2024-09-13T16:47:40.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.741+0000 7f98cbe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98cc0c1dc0 con 0x7f98cc14f880 2024-09-13T16:47:40.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.741+0000 7f98cbe00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98cc0c6bf0 con 0x7f98cc1562d0 2024-09-13T16:47:40.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.741+0000 7f98caa00640 1 -- >> v2:172.21.15.28:3300/0 conn(0x7f98cc14f880 msgr2=0x7f98cc155d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:47:40.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.741+0000 7f98caa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98cc14f880 0x7f98cc155d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:40.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.742+0000 7f98cb400640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f98cc1562d0 0x7f98cc15ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.742+0000 7f98ca000640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f98cc06fd80 0x7f98cc06b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:40808/0 (socket says 172.21.15.28:40808) 2024-09-13T16:47:40.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.742+0000 7f98ca000640 1 -- 172.21.15.28:0/3812990292 learned_addr learned my addr 172.21.15.28:0/3812990292 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:47:40.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.742+0000 7f98cb400640 1 -- 172.21.15.28:0/3812990292 >> v2:172.21.15.28:3300/0 conn(0x7f98cc14f880 msgr2=0x7f98cc155d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:40.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.742+0000 7f98cb400640 1 --2- 172.21.15.28:0/3812990292 >> v2:172.21.15.28:3300/0 conn(0x7f98cc14f880 0x7f98cc155d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.742+0000 7f98cb400640 1 -- 172.21.15.28:0/3812990292 >> v1:172.21.15.191:6789/0 conn(0x7f98cc06fd80 legacy=0x7f98cc06b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.742+0000 7f98cb400640 1 -- 172.21.15.28:0/3812990292 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc0c6630 con 0x7f98cc1562d0 2024-09-13T16:47:40.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.742+0000 7f98c9600640 1 -- 172.21.15.28:0/3812990292 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335547997 0 0) 0x7f98cc0c6350 con 0x7f98cc06fd80 2024-09-13T16:47:40.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.743+0000 7f98cb400640 1 --2- 172.21.15.28:0/3812990292 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1562d0 0x7f98cc15ff00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f98cc06af40 tx=0x7f98c402a730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96c0831016b978cd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:40.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.743+0000 7f98c9600640 1 -- 172.21.15.28:0/3812990292 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4007740 con 0x7f98cc1562d0 2024-09-13T16:47:40.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.743+0000 7f98c9600640 1 -- 172.21.15.28:0/3812990292 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c402d050 con 0x7f98cc1562d0 2024-09-13T16:47:40.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.743+0000 7f98c9600640 1 -- 172.21.15.28:0/3812990292 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4007ac0 con 0x7f98cc1562d0 2024-09-13T16:47:40.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.744+0000 7f98cbe00640 1 -- 172.21.15.28:0/3812990292 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1562d0 msgr2=0x7f98cc15ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.744+0000 7f98cbe00640 1 --2- 172.21.15.28:0/3812990292 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1562d0 0x7f98cc15ff00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f98cc06af40 tx=0x7f98c402a730 comp rx=0 tx=0).stop 2024-09-13T16:47:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.744+0000 7f98cbe00640 1 -- 172.21.15.28:0/3812990292 shutdown_connections 2024-09-13T16:47:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.744+0000 7f98cbe00640 1 --2- 172.21.15.28:0/3812990292 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1562d0 0x7f98cc15ff00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.744+0000 7f98cbe00640 1 --2- 172.21.15.28:0/3812990292 >> v2:172.21.15.28:3300/0 conn(0x7f98cc14f880 0x7f98cc155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.744+0000 7f98cbe00640 1 -- 172.21.15.28:0/3812990292 >> 172.21.15.28:0/3812990292 conn(0x7f98cc0ca2f0 msgr2=0x7f98cc0ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.744+0000 7f98cbe00640 1 -- 172.21.15.28:0/3812990292 shutdown_connections 2024-09-13T16:47:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.744+0000 7f98cbe00640 1 -- 172.21.15.28:0/3812990292 wait complete. 2024-09-13T16:47:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.745+0000 7f98cbe00640 1 Processor -- start 2024-09-13T16:47:40.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.746+0000 7f98cbe00640 1 -- start start 2024-09-13T16:47:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.747+0000 7f98cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98cc06fd80 0x7f98cc141d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.747+0000 7f98cbe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98cc14f880 0x7f98cc13f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.747+0000 7f98cbe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1562d0 0x7f98cc13f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.747+0000 7f98cbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc142950 con 0x7f98cc06fd80 2024-09-13T16:47:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.747+0000 7f98cbe00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc13e8c0 con 0x7f98cc1562d0 2024-09-13T16:47:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.747+0000 7f98cbe00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc13eba0 con 0x7f98cc14f880 2024-09-13T16:47:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.747+0000 7f98caa00640 1 -- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98cc06fd80 msgr2=0x7f98cc141d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:47:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.747+0000 7f98caa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98cc06fd80 0x7f98cc141d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.747+0000 7f98cb400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1562d0 0x7f98cc13f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.747+0000 7f98cb400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1562d0 0x7f98cc13f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:38964/0 (socket says 172.21.15.28:38964) 2024-09-13T16:47:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.747+0000 7f98cb400640 1 -- 172.21.15.28:0/3343933995 learned_addr learned my addr 172.21.15.28:0/3343933995 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.747+0000 7f98ca000640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98cc14f880 0x7f98cc13f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.747+0000 7f98cb400640 1 -- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98cc14f880 msgr2=0x7f98cc13f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.747+0000 7f98cb400640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f98cc14f880 0x7f98cc13f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.747+0000 7f98cb400640 1 -- 172.21.15.28:0/3343933995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98cc06fd80 msgr2=0x7f98cc141d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:47:40.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.747+0000 7f98cb400640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98cc06fd80 0x7f98cc141d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.747+0000 7f98cb400640 1 -- 172.21.15.28:0/3343933995 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c4009a70 con 0x7f98cc1562d0 2024-09-13T16:47:40.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.748+0000 7f98cb400640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1562d0 0x7f98cc13f570 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009e50 tx=0x7f98c4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:40.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.748+0000 7f98c2a00640 1 -- 172.21.15.28:0/3343933995 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4031910 con 0x7f98cc1562d0 2024-09-13T16:47:40.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.748+0000 7f98c2a00640 1 -- 172.21.15.28:0/3343933995 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c4031a70 con 0x7f98cc1562d0 2024-09-13T16:47:40.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.748+0000 7f98c2a00640 1 -- 172.21.15.28:0/3343933995 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4031de0 con 0x7f98cc1562d0 2024-09-13T16:47:40.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.748+0000 7f98cbe00640 1 -- 172.21.15.28:0/3343933995 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc152dc0 con 0x7f98cc1562d0 2024-09-13T16:47:40.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.748+0000 7f98cbe00640 1 -- 172.21.15.28:0/3343933995 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98cc0c75c0 con 0x7f98cc1562d0 2024-09-13T16:47:40.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.749+0000 7f98cbe00640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1554b0 0x7f98cc153fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.749+0000 7f98cbe00640 1 -- 172.21.15.28:0/3343933995 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc066c20 con 0x7f98cc1554b0 2024-09-13T16:47:40.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.749+0000 7f98caa00640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1554b0 0x7f98cc153fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.750+0000 7f98caa00640 1 -- 172.21.15.28:0/3343933995 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f98cc0c73f0 con 0x7f98cc1554b0 2024-09-13T16:47:40.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.750+0000 7f98caa00640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1554b0 0x7f98cc153fb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00b450 tx=0x7f98bc00b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:40.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.750+0000 7f98c2a00640 1 -- 172.21.15.28:0/3343933995 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f98c403c420 con 0x7f98cc1562d0 2024-09-13T16:47:40.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.751+0000 7f98c2a00640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f98ac06abb0 0x7f98ac06d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.751+0000 7f98c2a00640 1 -- 172.21.15.28:0/3343933995 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc003dc0 con 0x7f98cc1554b0 2024-09-13T16:47:40.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.751+0000 7f98c2a00640 1 -- 172.21.15.28:0/3343933995 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f98c40aaf50 con 0x7f98cc1562d0 2024-09-13T16:47:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.751+0000 7f98c2a00640 1 -- 172.21.15.28:0/3343933995 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f98bc005070 con 0x7f98cc1554b0 2024-09-13T16:47:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.751+0000 7f98c2a00640 1 -- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1554b0 msgr2=0x7f98cc153fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.751+0000 7f98c2a00640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1554b0 0x7f98cc153fb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00b450 tx=0x7f98bc00b910 comp rx=0 tx=0).stop 2024-09-13T16:47:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.751+0000 7f98c2a00640 1 -- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1554b0 msgr2=0x7f98cc153fb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:47:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.751+0000 7f98c2a00640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1554b0 0x7f98cc153fb0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00b450 tx=0x7f98bc00b910 comp rx=0 tx=0).stop 2024-09-13T16:47:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.751+0000 7f98ca000640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f98ac06abb0 0x7f98ac06d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.752+0000 7f98ca000640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f98ac06abb0 0x7f98ac06d070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f98cc142850 tx=0x7f98b8007d50 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:40.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.758+0000 7f98cbe00640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1544f0 0x7f98cc154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.758+0000 7f98cbe00640 1 -- 172.21.15.28:0/3343933995 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc155890 con 0x7f98cc1544f0 2024-09-13T16:47:40.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.759+0000 7f98caa00640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1544f0 0x7f98cc154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.759+0000 7f98caa00640 1 -- 172.21.15.28:0/3343933995 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f98cc154e40 con 0x7f98cc1544f0 2024-09-13T16:47:40.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.759+0000 7f98caa00640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1544f0 0x7f98cc154900 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00b450 tx=0x7f98bc0059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.759+0000 7f98c2a00640 1 -- 172.21.15.28:0/3343933995 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc01f070 con 0x7f98cc1544f0 2024-09-13T16:47:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.760+0000 7f98c2a00640 1 -- 172.21.15.28:0/3343933995 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4780 (secure 0 0 0) 0x7f98bc005820 con 0x7f98cc1544f0 2024-09-13T16:47:40.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.760+0000 7f98c2a00640 1 -- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1544f0 msgr2=0x7f98cc154900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.760+0000 7f98c2a00640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1544f0 0x7f98cc154900 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00b450 tx=0x7f98bc0059d0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.760+0000 7f98c2a00640 1 -- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1544f0 msgr2=0x7f98cc154900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:47:40.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.760+0000 7f98c2a00640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1544f0 0x7f98cc154900 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00b450 tx=0x7f98bc0059d0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.762 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:47:40.762 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:47:40.762 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:47:40.762 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "leader", 2024-09-13T16:47:40.762 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 26, 2024-09-13T16:47:40.762 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:47:40.763 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:47:40.763 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:47:40.763 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:40.763 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 0, 2024-09-13T16:47:40.763 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:47:40.763 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:47:40.763 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:47:40.763 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:40.763 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:40.763 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:40.763 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:40.763 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:40.763 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:40.764 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:40.764 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:40.764 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:40.764 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:40.764 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:40.764 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:47:40.764 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:47:40.764 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:40.764 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:40.764 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:40.764 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:40.764 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:40.765 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:40.765 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:40.765 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:40.765 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:40.765 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:40.765 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:40.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.765 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:47:40.765 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:47:40.765 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:47:40.765 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:47:40.765 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:47:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:47:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:47:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:47:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:47:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:47:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:47:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:47:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:47:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:47:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:47:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:47:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:47:40.766 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:47:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.767 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:47:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:47:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:47:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:47:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:47:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:47:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:47:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:40.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:47:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:47:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:47:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:47:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:47:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:47:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:47:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:47:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:47:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:40.772 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:47:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:47:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:47:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:40.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:47:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:47:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:47:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:47:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:47:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:40.774 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:47:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:47:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:47:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 3 2024-09-13T16:47:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:40.775 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:40.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.776 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:47:40.776 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:47:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.760+0000 7f98cbe00640 1 -- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f98ac06abb0 msgr2=0x7f98ac06d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.760+0000 7f98cbe00640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f98ac06abb0 0x7f98ac06d070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f98cc142850 tx=0x7f98b8007d50 comp rx=0 tx=0).stop 2024-09-13T16:47:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.760+0000 7f98cbe00640 1 -- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1562d0 msgr2=0x7f98cc13f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.760+0000 7f98cbe00640 1 --2- 172.21.15.28:0/3343933995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f98cc1562d0 0x7f98cc13f570 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009e50 tx=0x7f98c4004a40 comp rx=0 tx=0).stop 2024-09-13T16:47:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.760+0000 7f98cb400640 1 -- 172.21.15.28:0/3343933995 reap_dead start 2024-09-13T16:47:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.760+0000 7f98cb400640 1 -- 172.21.15.28:0/3343933995 reap_dead start 2024-09-13T16:47:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.760+0000 7f98cbe00640 1 -- 172.21.15.28:0/3343933995 shutdown_connections 2024-09-13T16:47:40.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.760+0000 7f98cbe00640 1 -- 172.21.15.28:0/3343933995 >> 172.21.15.28:0/3343933995 conn(0x7f98cc0ca2f0 msgr2=0x7f98cc0c22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.760+0000 7f98cbe00640 1 -- 172.21.15.28:0/3343933995 shutdown_connections 2024-09-13T16:47:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.760+0000 7f98cbe00640 1 -- 172.21.15.28:0/3343933995 wait complete. 2024-09-13T16:47:40.778 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-13T16:47:40.778 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-13T16:47:40.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T16:47:40.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.933+0000 7f3d49600640 1 Processor -- start 2024-09-13T16:47:40.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.934+0000 7f3d49600640 1 -- start start 2024-09-13T16:47:40.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.935+0000 7f3d49600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3d44152da0 0x7f3d440cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.935+0000 7f3d49600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d440ce7c0 con 0x7f3d440cadd0 2024-09-13T16:47:40.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.935+0000 7f3d49600640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d440cf060 con 0x7f3d440c9500 2024-09-13T16:47:40.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.935+0000 7f3d49600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d440c5060 con 0x7f3d44152da0 2024-09-13T16:47:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.935+0000 7f3d43400640 1 -- >> v1:172.21.15.28:6789/0 conn(0x7f3d440cadd0 legacy=0x7f3d44158b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.28:6789/0 2024-09-13T16:47:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.935+0000 7f3d43e00640 1 -- >> v2:172.21.15.28:3300/0 conn(0x7f3d44152da0 msgr2=0x7f3d440cbd20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:47:40.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.935+0000 7f3d43e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3d44152da0 0x7f3d440cbd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.935+0000 7f3d48c00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f3d440c9500 0x7f3d4406aa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:40810/0 (socket says 172.21.15.28:40810) 2024-09-13T16:47:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.935+0000 7f3d48c00640 1 -- 172.21.15.28:0/3011108216 learned_addr learned my addr 172.21.15.28:0/3011108216 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:47:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d42a00640 1 -- 172.21.15.28:0/3011108216 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569910442 0 0) 0x7f3d440cf060 con 0x7f3d440c9500 2024-09-13T16:47:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d42a00640 1 -- 172.21.15.28:0/3011108216 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d28003520 con 0x7f3d440c9500 2024-09-13T16:47:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d42a00640 1 -- 172.21.15.28:0/3011108216 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2861597166 0 0) 0x7f3d28003520 con 0x7f3d440c9500 2024-09-13T16:47:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d42a00640 1 -- 172.21.15.28:0/3011108216 >> v2:172.21.15.28:3300/0 conn(0x7f3d44152da0 msgr2=0x7f3d440cbd20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d42a00640 1 --2- 172.21.15.28:0/3011108216 >> v2:172.21.15.28:3300/0 conn(0x7f3d44152da0 0x7f3d440cbd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d42a00640 1 -- 172.21.15.28:0/3011108216 >> v1:172.21.15.28:6789/0 conn(0x7f3d440cadd0 legacy=0x7f3d44158b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d42a00640 1 -- 172.21.15.28:0/3011108216 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d440c9d00 con 0x7f3d440c9500 2024-09-13T16:47:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d42a00640 1 -- 172.21.15.28:0/3011108216 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1654382301 0 0) 0x7f3d3c0019f0 con 0x7f3d440c9500 2024-09-13T16:47:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d42a00640 1 -- 172.21.15.28:0/3011108216 >> v1:172.21.15.191:6789/0 conn(0x7f3d440c9500 legacy=0x7f3d4406aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d42a00640 1 --2- 172.21.15.28:0/3011108216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d28003970 0x7f3d28003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d42a00640 1 --2- 172.21.15.28:0/3011108216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d280047c0 0x7f3d28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d42a00640 1 --2- 172.21.15.28:0/3011108216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d280052a0 0x7f3d2800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d42a00640 1 -- 172.21.15.28:0/3011108216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2800c160 con 0x7f3d28003970 2024-09-13T16:47:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d42a00640 1 -- 172.21.15.28:0/3011108216 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2800c2d0 con 0x7f3d280052a0 2024-09-13T16:47:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d42a00640 1 -- 172.21.15.28:0/3011108216 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2800c490 con 0x7f3d280047c0 2024-09-13T16:47:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d43e00640 1 -- 172.21.15.28:0/3011108216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d28003970 msgr2=0x7f3d28003d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:47:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d43e00640 1 --2- 172.21.15.28:0/3011108216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d28003970 0x7f3d28003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.936+0000 7f3d43400640 1 --2- 172.21.15.28:0/3011108216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d280047c0 0x7f3d28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.937+0000 7f3d48c00640 1 --2- 172.21.15.28:0/3011108216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d280052a0 0x7f3d2800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.937+0000 7f3d43400640 1 -- 172.21.15.28:0/3011108216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d280052a0 msgr2=0x7f3d2800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:47:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.937+0000 7f3d43400640 1 --2- 172.21.15.28:0/3011108216 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d280052a0 0x7f3d2800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.937+0000 7f3d43400640 1 -- 172.21.15.28:0/3011108216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d28003970 msgr2=0x7f3d28003d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:47:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.937+0000 7f3d43400640 1 --2- 172.21.15.28:0/3011108216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d28003970 0x7f3d28003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.937+0000 7f3d43400640 1 -- 172.21.15.28:0/3011108216 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d28003520 con 0x7f3d280047c0 2024-09-13T16:47:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.937+0000 7f3d48c00640 1 -- 172.21.15.28:0/3011108216 reap_dead start 2024-09-13T16:47:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.938+0000 7f3d43400640 1 --2- 172.21.15.28:0/3011108216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d280047c0 0x7f3d28004c40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3d30003980 tx=0x7f3d280057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1b988b49099494a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.938+0000 7f3d42a00640 1 -- 172.21.15.28:0/3011108216 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d30003bb0 con 0x7f3d280047c0 2024-09-13T16:47:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.938+0000 7f3d42a00640 1 -- 172.21.15.28:0/3011108216 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d30003d10 con 0x7f3d280047c0 2024-09-13T16:47:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.938+0000 7f3d42a00640 1 -- 172.21.15.28:0/3011108216 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3002ab20 con 0x7f3d280047c0 2024-09-13T16:47:40.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.939+0000 7f3d49600640 1 -- 172.21.15.28:0/3011108216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d280047c0 msgr2=0x7f3d28004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.939+0000 7f3d49600640 1 --2- 172.21.15.28:0/3011108216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d280047c0 0x7f3d28004c40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3d30003980 tx=0x7f3d280057f0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.939+0000 7f3d49600640 1 -- 172.21.15.28:0/3011108216 shutdown_connections 2024-09-13T16:47:40.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.939+0000 7f3d49600640 1 --2- 172.21.15.28:0/3011108216 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d280047c0 0x7f3d28004c40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.939+0000 7f3d49600640 1 -- 172.21.15.28:0/3011108216 >> 172.21.15.28:0/3011108216 conn(0x7f3d4406c180 msgr2=0x7f3d440715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:40.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.939+0000 7f3d49600640 1 -- 172.21.15.28:0/3011108216 shutdown_connections 2024-09-13T16:47:40.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.939+0000 7f3d49600640 1 -- 172.21.15.28:0/3011108216 wait complete. 2024-09-13T16:47:40.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.941+0000 7f3d49600640 1 Processor -- start 2024-09-13T16:47:40.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.941+0000 7f3d49600640 1 -- start start 2024-09-13T16:47:40.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.942+0000 7f3d49600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d280047c0 0x7f3d440cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.942+0000 7f3d49600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d44155350 0x7f3d441f1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.942+0000 7f3d49600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d441ee850 0x7f3d441f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.942+0000 7f3d49600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d440c8060 con 0x7f3d280047c0 2024-09-13T16:47:40.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.942+0000 7f3d49600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d44154b00 con 0x7f3d44155350 2024-09-13T16:47:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.942+0000 7f3d49600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d44154ce0 con 0x7f3d441ee850 2024-09-13T16:47:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.942+0000 7f3d43e00640 1 -- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d280047c0 msgr2=0x7f3d440cda00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:3300/0 2024-09-13T16:47:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.942+0000 7f3d43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d280047c0 0x7f3d440cda00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:47:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.942+0000 7f3d48c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d441ee850 0x7f3d441f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.942+0000 7f3d48c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d441ee850 0x7f3d441f1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:35250/0 (socket says 172.21.15.28:35250) 2024-09-13T16:47:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.942+0000 7f3d48c00640 1 -- 172.21.15.28:0/1990643707 learned_addr learned my addr 172.21.15.28:0/1990643707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:47:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.942+0000 7f3d43400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d44155350 0x7f3d441f1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.942+0000 7f3d48c00640 1 -- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d44155350 msgr2=0x7f3d441f1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.942+0000 7f3d48c00640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d44155350 0x7f3d441f1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.942+0000 7f3d48c00640 1 -- 172.21.15.28:0/1990643707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d280047c0 msgr2=0x7f3d440cda00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:47:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.942+0000 7f3d48c00640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d280047c0 0x7f3d440cda00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.942+0000 7f3d48c00640 1 -- 172.21.15.28:0/1990643707 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d44156010 con 0x7f3d441ee850 2024-09-13T16:47:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.943+0000 7f3d48c00640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d441ee850 0x7f3d441f1c60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c007970 tx=0x7f3d3c00a600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.943+0000 7f3d3be00640 1 -- 172.21.15.28:0/1990643707 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c00ac30 con 0x7f3d441ee850 2024-09-13T16:47:40.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.943+0000 7f3d3be00640 1 -- 172.21.15.28:0/1990643707 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3c00ad90 con 0x7f3d441ee850 2024-09-13T16:47:40.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.943+0000 7f3d49600640 1 -- 172.21.15.28:0/1990643707 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d441562f0 con 0x7f3d441ee850 2024-09-13T16:47:40.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.943+0000 7f3d3be00640 1 -- 172.21.15.28:0/1990643707 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c0135f0 con 0x7f3d441ee850 2024-09-13T16:47:40.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.943+0000 7f3d49600640 1 -- 172.21.15.28:0/1990643707 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d440ce2b0 con 0x7f3d441ee850 2024-09-13T16:47:40.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.944+0000 7f3d49600640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d440c88f0 0x7f3d4414b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.944+0000 7f3d49600640 1 -- 172.21.15.28:0/1990643707 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d44072440 con 0x7f3d440c88f0 2024-09-13T16:47:40.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.944+0000 7f3d43e00640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d440c88f0 0x7f3d4414b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.944+0000 7f3d43e00640 1 -- 172.21.15.28:0/1990643707 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3d440cecf0 con 0x7f3d440c88f0 2024-09-13T16:47:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.945+0000 7f3d43e00640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d440c88f0 0x7f3d4414b7c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3d340038b0 tx=0x7f3d3400b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.945+0000 7f3d3be00640 1 -- 172.21.15.28:0/1990643707 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d34003dd0 con 0x7f3d440c88f0 2024-09-13T16:47:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.945+0000 7f3d3be00640 1 -- 172.21.15.28:0/1990643707 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3d3c013750 con 0x7f3d441ee850 2024-09-13T16:47:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.946+0000 7f3d3be00640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3d2406acd0 0x7f3d2406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.946+0000 7f3d3be00640 1 -- 172.21.15.28:0/1990643707 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3d3c089970 con 0x7f3d441ee850 2024-09-13T16:47:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.946+0000 7f3d43400640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3d2406acd0 0x7f3d2406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.946+0000 7f3d3be00640 1 -- 172.21.15.28:0/1990643707 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f3d34005070 con 0x7f3d440c88f0 2024-09-13T16:47:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.946+0000 7f3d3be00640 1 -- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d440c88f0 msgr2=0x7f3d4414b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.946+0000 7f3d3be00640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d440c88f0 0x7f3d4414b7c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3d340038b0 tx=0x7f3d3400b8c0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.946+0000 7f3d3be00640 1 -- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d440c88f0 msgr2=0x7f3d4414b7c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:47:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.946+0000 7f3d3be00640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d440c88f0 0x7f3d4414b7c0 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3d340038b0 tx=0x7f3d3400b8c0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.947+0000 7f3d43400640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3d2406acd0 0x7f3d2406d190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3d30032000 tx=0x7f3d30008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.953+0000 7f3d49600640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d4414bd00 0x7f3d441534f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:47:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.953+0000 7f3d49600640 1 -- 172.21.15.28:0/1990643707 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d440cea10 con 0x7f3d4414bd00 2024-09-13T16:47:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.953+0000 7f3d43e00640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d4414bd00 0x7f3d441534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:47:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.953+0000 7f3d43e00640 1 -- 172.21.15.28:0/1990643707 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3d440cf4d0 con 0x7f3d4414bd00 2024-09-13T16:47:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.954+0000 7f3d43e00640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d4414bd00 0x7f3d441534f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3d340038b0 tx=0x7f3d340059e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:47:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.954+0000 7f3d3be00640 1 -- 172.21.15.28:0/1990643707 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3401f070 con 0x7f3d4414bd00 2024-09-13T16:47:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.954+0000 7f3d3be00640 1 -- 172.21.15.28:0/1990643707 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+5087 (secure 0 0 0) 0x7f3d3401a070 con 0x7f3d4414bd00 2024-09-13T16:47:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.954+0000 7f3d3be00640 1 -- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d4414bd00 msgr2=0x7f3d441534f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.954+0000 7f3d3be00640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d4414bd00 0x7f3d441534f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3d340038b0 tx=0x7f3d340059e0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.954+0000 7f3d3be00640 1 -- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d4414bd00 msgr2=0x7f3d441534f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:47:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.954+0000 7f3d3be00640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d4414bd00 0x7f3d441534f0 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3d340038b0 tx=0x7f3d340059e0 comp rx=0 tx=0).stop 2024-09-13T16:47:40.957 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:47:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:47:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:47:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:47:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 26, 2024-09-13T16:47:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:47:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:47:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:47:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 0, 2024-09-13T16:47:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:47:40.957 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:47:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:47:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:47:40.958 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:47:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.959 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:47:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:47:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:47:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:47:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:47:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:47:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:47:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:47:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:47:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:47:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:47:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:47:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:47:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:47:40.960 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:47:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:47:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:47:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:47:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:47:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:47:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:47:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:47:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:47:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:47:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:47:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:47:40.961 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:47:40.962 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:40.962 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:47:40.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.962 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:47:40.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.962 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:47:40.962 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:47:40.962 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:40.962 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:40.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:40.962 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:47:40.962 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:40.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:47:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:47:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:47:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:40.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:47:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:47:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:47:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:47:40.964 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:47:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:47:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:47:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:47:40.965 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:47:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:47:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:47:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:47:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:47:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:47:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:47:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:40.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:47:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:47:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:47:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:47:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:47:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:47:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:47:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:40.967 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:47:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:47:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 5 2024-09-13T16:47:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:47:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.968 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 6 2024-09-13T16:47:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:47:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x2f018fb87aa4aafe", 2024-09-13T16:47:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "luminous", 2024-09-13T16:47:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:47:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:40.969 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 5 2024-09-13T16:47:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:47:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": [ 2024-09-13T16:47:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:47:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:47:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:47:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:47:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:47:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:47:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:47:40.970 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:47:40.970 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:47:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.955+0000 7f3d49600640 1 -- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3d2406acd0 msgr2=0x7f3d2406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.955+0000 7f3d49600640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f3d2406acd0 0x7f3d2406d190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3d30032000 tx=0x7f3d30008040 comp rx=0 tx=0).stop 2024-09-13T16:47:40.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.955+0000 7f3d49600640 1 -- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d441ee850 msgr2=0x7f3d441f1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:47:40.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.955+0000 7f3d49600640 1 --2- 172.21.15.28:0/1990643707 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3d441ee850 0x7f3d441f1c60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c007970 tx=0x7f3d3c00a600 comp rx=0 tx=0).stop 2024-09-13T16:47:40.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.955+0000 7f3d48c00640 1 -- 172.21.15.28:0/1990643707 reap_dead start 2024-09-13T16:47:40.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.955+0000 7f3d48c00640 1 -- 172.21.15.28:0/1990643707 reap_dead start 2024-09-13T16:47:40.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.955+0000 7f3d49600640 1 -- 172.21.15.28:0/1990643707 shutdown_connections 2024-09-13T16:47:40.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.955+0000 7f3d49600640 1 -- 172.21.15.28:0/1990643707 >> 172.21.15.28:0/1990643707 conn(0x7f3d4406c180 msgr2=0x7f3d440cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:47:40.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.955+0000 7f3d49600640 1 -- 172.21.15.28:0/1990643707 shutdown_connections 2024-09-13T16:47:40.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:47:40.955+0000 7f3d49600640 1 -- 172.21.15.28:0/1990643707 wait complete. 2024-09-13T16:47:40.972 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-13T16:47:40.973 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-13T16:47:45.407 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:47:45.407 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 readers = 5272209.50 kB/sec 2024-09-13T16:47:45.407 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 readers = 5269803.30 kB/sec 2024-09-13T16:47:45.407 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 5272209.50 kB/sec 2024-09-13T16:47:45.407 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 5272209.50 kB/sec 2024-09-13T16:47:45.407 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 5272209.50 kB/sec 2024-09-13T16:47:45.407 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 10485760.00 kB 2024-09-13T16:47:50.431 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:47:50.432 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 re-readers = 3577633.00 kB/sec 2024-09-13T16:47:50.432 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 re-readers = 3576639.78 kB/sec 2024-09-13T16:47:50.432 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 3577633.00 kB/sec 2024-09-13T16:47:50.432 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 3577633.00 kB/sec 2024-09-13T16:47:50.432 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 3577633.00 kB/sec 2024-09-13T16:47:50.432 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 10485760.00 kB 2024-09-13T16:48:00.105 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:48:00.105 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:48:00.105 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-09-13T16:48:00.105 INFO:tasks.workunit.client.0.smithi028.stdout:iozone test complete. 2024-09-13T16:48:00.108 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T16:48:00.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T16:48:00.177 INFO:tasks.workunit:Stopping ['suites/iozone.sh'] on client.0... 2024-09-13T16:48:00.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-13T16:48:00.973 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-13T16:48:00.973 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-13T16:48:00.974 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-13T16:48:00.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-13T16:48:00.978 INFO:tasks.ceph.mon.a:Started 2024-09-13T16:48:00.978 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T16:48:00.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:48:01.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.133+0000 7feb00200640 1 Processor -- start 2024-09-13T16:48:01.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.133+0000 7feb00200640 1 -- start start 2024-09-13T16:48:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.135+0000 7feb00200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaf80c96f0 0x7feaf80c7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.135+0000 7feb00200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf8159020 con 0x7feaf806b000 2024-09-13T16:48:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.135+0000 7feb00200640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf8159300 con 0x7feaf80c9310 2024-09-13T16:48:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.135+0000 7feb00200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf80c7550 con 0x7feaf80c96f0 2024-09-13T16:48:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.135+0000 7feafe200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7feaf806b000 0x7feaf806df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50522/0 (socket says 172.21.15.28:50522) 2024-09-13T16:48:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.135+0000 7feafe200640 1 -- 172.21.15.28:0/205284710 learned_addr learned my addr 172.21.15.28:0/205284710 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.135+0000 7feafce00640 1 --2- 172.21.15.28:0/205284710 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaf80c96f0 0x7feaf80c7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.135+0000 7feaf7e00640 1 -- 172.21.15.28:0/205284710 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392281311 0 0) 0x7feaf8159300 con 0x7feaf80c9310 2024-09-13T16:48:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.136+0000 7feaf7e00640 1 -- 172.21.15.28:0/205284710 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feae0003520 con 0x7feaf80c9310 2024-09-13T16:48:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.136+0000 7feafce00640 1 -- 172.21.15.28:0/205284710 >> v1:172.21.15.191:6789/0 conn(0x7feaf80c9310 legacy=0x7feaf8155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.136+0000 7feafce00640 1 -- 172.21.15.28:0/205284710 >> v1:172.21.15.28:6789/0 conn(0x7feaf806b000 legacy=0x7feaf806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.136+0000 7feafce00640 1 -- 172.21.15.28:0/205284710 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf81595e0 con 0x7feaf80c96f0 2024-09-13T16:48:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.137+0000 7feafce00640 1 --2- 172.21.15.28:0/205284710 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaf80c96f0 0x7feaf80c7010 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feae40097d0 tx=0x7feae402a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=140c694025765b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:01.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.137+0000 7feaf7e00640 1 -- 172.21.15.28:0/205284710 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae4032440 con 0x7feaf80c96f0 2024-09-13T16:48:01.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.137+0000 7feaf7e00640 1 -- 172.21.15.28:0/205284710 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae40325a0 con 0x7feaf80c96f0 2024-09-13T16:48:01.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.137+0000 7feaf7e00640 1 -- 172.21.15.28:0/205284710 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae4032910 con 0x7feaf80c96f0 2024-09-13T16:48:01.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.137+0000 7feb00200640 1 -- 172.21.15.28:0/205284710 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaf80c96f0 msgr2=0x7feaf80c7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:01.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.137+0000 7feb00200640 1 --2- 172.21.15.28:0/205284710 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaf80c96f0 0x7feaf80c7010 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feae40097d0 tx=0x7feae402a290 comp rx=0 tx=0).stop 2024-09-13T16:48:01.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.138+0000 7feb00200640 1 -- 172.21.15.28:0/205284710 shutdown_connections 2024-09-13T16:48:01.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.138+0000 7feb00200640 1 --2- 172.21.15.28:0/205284710 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaf80c96f0 0x7feaf80c7010 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:01.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.138+0000 7feb00200640 1 -- 172.21.15.28:0/205284710 >> 172.21.15.28:0/205284710 conn(0x7feaf806fc30 msgr2=0x7feaf8070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:01.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.138+0000 7feb00200640 1 -- 172.21.15.28:0/205284710 shutdown_connections 2024-09-13T16:48:01.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.138+0000 7feb00200640 1 -- 172.21.15.28:0/205284710 wait complete. 2024-09-13T16:48:01.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.139+0000 7feb00200640 1 Processor -- start 2024-09-13T16:48:01.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.139+0000 7feb00200640 1 -- start start 2024-09-13T16:48:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.140+0000 7feb00200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feaf806b000 0x7feaf81473d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.140+0000 7feb00200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaf80c9310 0x7feaf81544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.140+0000 7feb00200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feaf80c96f0 0x7feaf8154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:01.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.140+0000 7feb00200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf8143da0 con 0x7feaf80c96f0 2024-09-13T16:48:01.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.140+0000 7feb00200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf8154f50 con 0x7feaf806b000 2024-09-13T16:48:01.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.140+0000 7feb00200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf8144fd0 con 0x7feaf80c9310 2024-09-13T16:48:01.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.141+0000 7feafe200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feaf80c96f0 0x7feaf8154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:01.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.141+0000 7feafe200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feaf80c96f0 0x7feaf8154a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60190/0 (socket says 172.21.15.28:60190) 2024-09-13T16:48:01.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.141+0000 7feafe200640 1 -- 172.21.15.28:0/1997274300 learned_addr learned my addr 172.21.15.28:0/1997274300 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:01.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.141+0000 7feafd800640 1 --2- 172.21.15.28:0/1997274300 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feaf806b000 0x7feaf81473d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:01.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.141+0000 7feafce00640 1 --2- 172.21.15.28:0/1997274300 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaf80c9310 0x7feaf81544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:01.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.141+0000 7feafe200640 1 -- 172.21.15.28:0/1997274300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feaf80c96f0 msgr2=0x7feaf8154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-13T16:48:01.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.141+0000 7feafe200640 1 -- 172.21.15.28:0/1997274300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feaf80c96f0 msgr2=0x7feaf8154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T16:48:01.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.141+0000 7feafe200640 1 --2- 172.21.15.28:0/1997274300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feaf80c96f0 0x7feaf8154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:48:01.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.141+0000 7feafe200640 1 --2- 172.21.15.28:0/1997274300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feaf80c96f0 0x7feaf8154a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:48:01.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.141+0000 7feafd800640 1 -- 172.21.15.28:0/1997274300 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaf80c9310 msgr2=0x7feaf81544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.141+0000 7feafd800640 1 --2- 172.21.15.28:0/1997274300 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaf80c9310 0x7feaf81544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.141+0000 7feafd800640 1 -- 172.21.15.28:0/1997274300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feaf80c96f0 msgr2=0x7feaf8154a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:48:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.141+0000 7feafd800640 1 --2- 172.21.15.28:0/1997274300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feaf80c96f0 0x7feaf8154a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.141+0000 7feafd800640 1 -- 172.21.15.28:0/1997274300 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae4009400 con 0x7feaf806b000 2024-09-13T16:48:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.141+0000 7feafd800640 1 --2- 172.21.15.28:0/1997274300 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feaf806b000 0x7feaf81473d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7feaec00c430 tx=0x7feaec00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.142+0000 7feaf5600640 1 -- 172.21.15.28:0/1997274300 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaec0052d0 con 0x7feaf806b000 2024-09-13T16:48:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.142+0000 7feaf5600640 1 -- 172.21.15.28:0/1997274300 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaec005430 con 0x7feaf806b000 2024-09-13T16:48:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.142+0000 7feaf5600640 1 -- 172.21.15.28:0/1997274300 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaec003830 con 0x7feaf806b000 2024-09-13T16:48:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.142+0000 7feb00200640 1 -- 172.21.15.28:0/1997274300 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf81449f0 con 0x7feaf806b000 2024-09-13T16:48:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.142+0000 7feb00200640 1 -- 172.21.15.28:0/1997274300 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf81556e0 con 0x7feaf806b000 2024-09-13T16:48:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.143+0000 7feb00200640 1 -- 172.21.15.28:0/1997274300 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf8159300 con 0x7feaf806b000 2024-09-13T16:48:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.144+0000 7feaf5600640 1 -- 172.21.15.28:0/1997274300 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7feaec003990 con 0x7feaf806b000 2024-09-13T16:48:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.145+0000 7feaf5600640 1 --2- 172.21.15.28:0/1997274300 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fead806ac30 0x7fead806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.145+0000 7feaf5600640 1 -- 172.21.15.28:0/1997274300 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7feaec0889c0 con 0x7feaf806b000 2024-09-13T16:48:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.145+0000 7feafce00640 1 --2- 172.21.15.28:0/1997274300 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fead806ac30 0x7fead806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.146+0000 7feafce00640 1 --2- 172.21.15.28:0/1997274300 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fead806ac30 0x7fead806d0f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7feae4034000 tx=0x7feae402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:01.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.149+0000 7feaf5600640 1 -- 172.21.15.28:0/1997274300 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7feaec053b30 con 0x7feaf806b000 2024-09-13T16:48:01.212 DEBUG:teuthology.parallel:result is None 2024-09-13T16:48:01.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T16:48:01.278 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T16:48:01.278 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-13T16:48:01.288 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-13T16:48:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.395+0000 7feb00200640 1 -- 172.21.15.28:0/1997274300 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7feaf80c77a0 con 0x7feaf806b000 2024-09-13T16:48:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.396+0000 7feaf5600640 1 -- 172.21.15.28:0/1997274300 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7feaec017080 con 0x7feaf806b000 2024-09-13T16:48:01.398 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":26,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:48:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.398+0000 7feb00200640 1 -- 172.21.15.28:0/1997274300 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fead806ac30 msgr2=0x7fead806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.398+0000 7feb00200640 1 --2- 172.21.15.28:0/1997274300 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fead806ac30 0x7fead806d0f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7feae4034000 tx=0x7feae402e000 comp rx=0 tx=0).stop 2024-09-13T16:48:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.398+0000 7feb00200640 1 -- 172.21.15.28:0/1997274300 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feaf806b000 msgr2=0x7feaf81473d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.398+0000 7feb00200640 1 --2- 172.21.15.28:0/1997274300 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feaf806b000 0x7feaf81473d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7feaec00c430 tx=0x7feaec00c8f0 comp rx=0 tx=0).stop 2024-09-13T16:48:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.398+0000 7feb00200640 1 -- 172.21.15.28:0/1997274300 shutdown_connections 2024-09-13T16:48:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.398+0000 7feb00200640 1 --2- 172.21.15.28:0/1997274300 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fead806ac30 0x7fead806d0f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.398+0000 7feb00200640 1 --2- 172.21.15.28:0/1997274300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feaf80c96f0 0x7feaf8154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.398+0000 7feb00200640 1 --2- 172.21.15.28:0/1997274300 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaf80c9310 0x7feaf81544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.398+0000 7feb00200640 1 --2- 172.21.15.28:0/1997274300 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feaf806b000 0x7feaf81473d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.398+0000 7feb00200640 1 -- 172.21.15.28:0/1997274300 >> 172.21.15.28:0/1997274300 conn(0x7feaf806fc30 msgr2=0x7feaf8153010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.398+0000 7feb00200640 1 -- 172.21.15.28:0/1997274300 shutdown_connections 2024-09-13T16:48:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:01.398+0000 7feb00200640 1 -- 172.21.15.28:0/1997274300 wait complete. 2024-09-13T16:48:04.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:48:04.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:48:04.534 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:48:04.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.581+0000 7f4b85400640 1 Processor -- start 2024-09-13T16:48:04.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.582+0000 7f4b85400640 1 -- start start 2024-09-13T16:48:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.583+0000 7f4b85400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b80072170 0x7f4b800c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.583+0000 7f4b85400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4b800c6350 0x7f4b801559d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.583+0000 7f4b85400640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b800c42b0 con 0x7f4b8014fa80 2024-09-13T16:48:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.583+0000 7f4b85400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b800c36e0 con 0x7f4b800c6350 2024-09-13T16:48:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.583+0000 7f4b85400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b800c3980 con 0x7f4b80072170 2024-09-13T16:48:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.583+0000 7f4b7e800640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f4b8014fa80 0x7f4b800c6ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:60078/0 (socket says 172.21.15.28:60078) 2024-09-13T16:48:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.583+0000 7f4b77e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b80072170 0x7f4b800c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.583+0000 7f4b7e800640 1 -- 172.21.15.28:0/4251977168 learned_addr learned my addr 172.21.15.28:0/4251977168 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.583+0000 7f4b7f200640 1 --2- 172.21.15.28:0/4251977168 >> v2:172.21.15.28:3300/0 conn(0x7f4b800c6350 0x7f4b801559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.583+0000 7f4b7f200640 1 -- 172.21.15.28:0/4251977168 >> v2:172.21.15.28:3300/0 conn(0x7f4b800c6350 msgr2=0x7f4b801559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-13T16:48:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.583+0000 7f4b7f200640 1 -- 172.21.15.28:0/4251977168 >> v2:172.21.15.28:3300/0 conn(0x7f4b800c6350 msgr2=0x7f4b801559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:48:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.583+0000 7f4b7f200640 1 --2- 172.21.15.28:0/4251977168 >> v2:172.21.15.28:3300/0 conn(0x7f4b800c6350 0x7f4b801559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:48:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.584+0000 7f4b7f200640 1 --2- 172.21.15.28:0/4251977168 >> v2:172.21.15.28:3300/0 conn(0x7f4b800c6350 0x7f4b801559d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:48:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.584+0000 7f4b7de00640 1 -- 172.21.15.28:0/4251977168 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3302748604 0 0) 0x7f4b800c42b0 con 0x7f4b8014fa80 2024-09-13T16:48:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.584+0000 7f4b7de00640 1 -- 172.21.15.28:0/4251977168 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b64005730 con 0x7f4b8014fa80 2024-09-13T16:48:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.584+0000 7f4b77e00640 1 -- 172.21.15.28:0/4251977168 >> v2:172.21.15.28:3300/0 conn(0x7f4b800c6350 msgr2=0x7f4b801559d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:48:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.584+0000 7f4b77e00640 1 --2- 172.21.15.28:0/4251977168 >> v2:172.21.15.28:3300/0 conn(0x7f4b800c6350 0x7f4b801559d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.584+0000 7f4b77e00640 1 -- 172.21.15.28:0/4251977168 >> v1:172.21.15.191:6789/0 conn(0x7f4b8014fa80 legacy=0x7f4b800c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.584+0000 7f4b77e00640 1 -- 172.21.15.28:0/4251977168 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b800c3c60 con 0x7f4b80072170 2024-09-13T16:48:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.585+0000 7f4b77e00640 1 --2- 172.21.15.28:0/4251977168 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b80072170 0x7f4b800c1960 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4b680097a0 tx=0x7f4b6802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=955cae2119e79d4a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.585+0000 7f4b7de00640 1 -- 172.21.15.28:0/4251977168 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68032440 con 0x7f4b80072170 2024-09-13T16:48:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.585+0000 7f4b7de00640 1 -- 172.21.15.28:0/4251977168 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b680325a0 con 0x7f4b80072170 2024-09-13T16:48:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.585+0000 7f4b7de00640 1 -- 172.21.15.28:0/4251977168 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68032910 con 0x7f4b80072170 2024-09-13T16:48:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.586+0000 7f4b85400640 1 -- 172.21.15.28:0/4251977168 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b80072170 msgr2=0x7f4b800c1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.586+0000 7f4b85400640 1 --2- 172.21.15.28:0/4251977168 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b80072170 0x7f4b800c1960 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4b680097a0 tx=0x7f4b6802a180 comp rx=0 tx=0).stop 2024-09-13T16:48:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.586+0000 7f4b85400640 1 -- 172.21.15.28:0/4251977168 shutdown_connections 2024-09-13T16:48:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.586+0000 7f4b85400640 1 --2- 172.21.15.28:0/4251977168 >> v2:172.21.15.28:3300/0 conn(0x7f4b800c6350 0x7f4b801559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:04.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.586+0000 7f4b85400640 1 --2- 172.21.15.28:0/4251977168 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b80072170 0x7f4b800c1960 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.586+0000 7f4b85400640 1 -- 172.21.15.28:0/4251977168 >> 172.21.15.28:0/4251977168 conn(0x7f4b80069e80 msgr2=0x7f4b8006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.586+0000 7f4b85400640 1 -- 172.21.15.28:0/4251977168 shutdown_connections 2024-09-13T16:48:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.586+0000 7f4b85400640 1 -- 172.21.15.28:0/4251977168 wait complete. 2024-09-13T16:48:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.587+0000 7f4b85400640 1 Processor -- start 2024-09-13T16:48:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.588+0000 7f4b85400640 1 -- start start 2024-09-13T16:48:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.588+0000 7f4b85400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b80072170 0x7f4b80152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.588+0000 7f4b85400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b800c6350 0x7f4b801e0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b85400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b8014fa80 0x7f4b80153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b85400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b80151ba0 con 0x7f4b80072170 2024-09-13T16:48:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b85400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b801e1ce0 con 0x7f4b8014fa80 2024-09-13T16:48:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b85400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b801529f0 con 0x7f4b800c6350 2024-09-13T16:48:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b7e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b80072170 0x7f4b80152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b77e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b800c6350 0x7f4b801e0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b7f200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b8014fa80 0x7f4b80153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b7e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b80072170 0x7f4b80152cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60198/0 (socket says 172.21.15.28:60198) 2024-09-13T16:48:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b7e800640 1 -- 172.21.15.28:0/4230851253 learned_addr learned my addr 172.21.15.28:0/4230851253 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b7e800640 1 -- 172.21.15.28:0/4230851253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b80072170 msgr2=0x7f4b80152cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-13T16:48:04.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b7e800640 1 -- 172.21.15.28:0/4230851253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b80072170 msgr2=0x7f4b80152cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T16:48:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b7e800640 1 --2- 172.21.15.28:0/4230851253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b80072170 0x7f4b80152cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:48:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b7e800640 1 --2- 172.21.15.28:0/4230851253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b80072170 0x7f4b80152cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:48:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b77e00640 1 -- 172.21.15.28:0/4230851253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b8014fa80 msgr2=0x7f4b80153670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b77e00640 1 --2- 172.21.15.28:0/4230851253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b8014fa80 0x7f4b80153670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b77e00640 1 -- 172.21.15.28:0/4230851253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b80072170 msgr2=0x7f4b80152cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:48:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b77e00640 1 --2- 172.21.15.28:0/4230851253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b80072170 0x7f4b80152cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.589+0000 7f4b77e00640 1 -- 172.21.15.28:0/4230851253 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b68009400 con 0x7f4b800c6350 2024-09-13T16:48:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.590+0000 7f4b77e00640 1 --2- 172.21.15.28:0/4230851253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b800c6350 0x7f4b801e0b90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4b800c3ac0 tx=0x7f4b680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.590+0000 7f4b76a00640 1 -- 172.21.15.28:0/4230851253 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68030b80 con 0x7f4b800c6350 2024-09-13T16:48:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.590+0000 7f4b76a00640 1 -- 172.21.15.28:0/4230851253 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b68030d00 con 0x7f4b800c6350 2024-09-13T16:48:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.590+0000 7f4b76a00640 1 -- 172.21.15.28:0/4230851253 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6803c640 con 0x7f4b800c6350 2024-09-13T16:48:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.590+0000 7f4b85400640 1 -- 172.21.15.28:0/4230851253 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b80153230 con 0x7f4b800c6350 2024-09-13T16:48:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.590+0000 7f4b85400640 1 -- 172.21.15.28:0/4230851253 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b801e1a00 con 0x7f4b800c6350 2024-09-13T16:48:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.591+0000 7f4b85400640 1 -- 172.21.15.28:0/4230851253 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b80071eb0 con 0x7f4b800c6350 2024-09-13T16:48:04.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.592+0000 7f4b76a00640 1 -- 172.21.15.28:0/4230851253 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4b6803c7a0 con 0x7f4b800c6350 2024-09-13T16:48:04.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.593+0000 7f4b76a00640 1 --2- 172.21.15.28:0/4230851253 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4b5c06ac00 0x7f4b5c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:04.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.593+0000 7f4b76a00640 1 -- 172.21.15.28:0/4230851253 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4b680aa1c0 con 0x7f4b800c6350 2024-09-13T16:48:04.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.593+0000 7f4b7e800640 1 --2- 172.21.15.28:0/4230851253 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4b5c06ac00 0x7f4b5c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:04.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.594+0000 7f4b7e800640 1 --2- 172.21.15.28:0/4230851253 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4b5c06ac00 0x7f4b5c06d0c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4b70005880 tx=0x7f4b70009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:04.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.598+0000 7f4b76a00640 1 -- 172.21.15.28:0/4230851253 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4b68075330 con 0x7f4b800c6350 2024-09-13T16:48:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.840+0000 7f4b85400640 1 -- 172.21.15.28:0/4230851253 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4b800711f0 con 0x7f4b800c6350 2024-09-13T16:48:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.840+0000 7f4b76a00640 1 -- 172.21.15.28:0/4230851253 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1542 (secure 0 0 0) 0x7f4b68006e10 con 0x7f4b800c6350 2024-09-13T16:48:04.842 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":26,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:48:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.842+0000 7f4b85400640 1 -- 172.21.15.28:0/4230851253 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4b5c06ac00 msgr2=0x7f4b5c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.842+0000 7f4b85400640 1 --2- 172.21.15.28:0/4230851253 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4b5c06ac00 0x7f4b5c06d0c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4b70005880 tx=0x7f4b70009040 comp rx=0 tx=0).stop 2024-09-13T16:48:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.842+0000 7f4b85400640 1 -- 172.21.15.28:0/4230851253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b800c6350 msgr2=0x7f4b801e0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.842+0000 7f4b85400640 1 --2- 172.21.15.28:0/4230851253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b800c6350 0x7f4b801e0b90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4b800c3ac0 tx=0x7f4b680046c0 comp rx=0 tx=0).stop 2024-09-13T16:48:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.842+0000 7f4b85400640 1 -- 172.21.15.28:0/4230851253 shutdown_connections 2024-09-13T16:48:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.842+0000 7f4b85400640 1 --2- 172.21.15.28:0/4230851253 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4b5c06ac00 0x7f4b5c06d0c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.842+0000 7f4b85400640 1 --2- 172.21.15.28:0/4230851253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b8014fa80 0x7f4b80153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:04.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.842+0000 7f4b85400640 1 --2- 172.21.15.28:0/4230851253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4b800c6350 0x7f4b801e0b90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:04.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.842+0000 7f4b85400640 1 --2- 172.21.15.28:0/4230851253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b80072170 0x7f4b80152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:04.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.842+0000 7f4b85400640 1 -- 172.21.15.28:0/4230851253 >> 172.21.15.28:0/4230851253 conn(0x7f4b80069e80 msgr2=0x7f4b800c7610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:04.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.842+0000 7f4b85400640 1 -- 172.21.15.28:0/4230851253 shutdown_connections 2024-09-13T16:48:04.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:04.842+0000 7f4b85400640 1 -- 172.21.15.28:0/4230851253 wait complete. 2024-09-13T16:48:07.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:48:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.026+0000 7fae5b200640 1 Processor -- start 2024-09-13T16:48:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.027+0000 7fae5b200640 1 -- start start 2024-09-13T16:48:08.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.028+0000 7fae5b200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fae54150a70 0x7fae540c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.028+0000 7fae5b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fae54156c30 0x7fae54157040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.028+0000 7fae5b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae54159f70 con 0x7fae540c6720 2024-09-13T16:48:08.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.028+0000 7fae5b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae540c64f0 con 0x7fae54156c30 2024-09-13T16:48:08.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.028+0000 7fae5b200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae540c1ea0 con 0x7fae54150a70 2024-09-13T16:48:08.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.028+0000 7fae4be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fae540c6720 0x7fae540c4e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50532/0 (socket says 172.21.15.28:50532) 2024-09-13T16:48:08.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.028+0000 7fae4be00640 1 -- 172.21.15.28:0/3878470992 learned_addr learned my addr 172.21.15.28:0/3878470992 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:08.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.028+0000 7fae59200640 1 --2- 172.21.15.28:0/3878470992 >> v2:172.21.15.28:3300/0 conn(0x7fae54156c30 0x7fae54157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.028+0000 7fae53e00640 1 --2- 172.21.15.28:0/3878470992 >> v2:172.21.15.191:3300/0 conn(0x7fae54150a70 0x7fae540c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.029+0000 7fae59200640 1 -- 172.21.15.28:0/3878470992 >> v2:172.21.15.191:3300/0 conn(0x7fae54150a70 msgr2=0x7fae540c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:08.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.029+0000 7fae59200640 1 --2- 172.21.15.28:0/3878470992 >> v2:172.21.15.191:3300/0 conn(0x7fae54150a70 0x7fae540c1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.029+0000 7fae59200640 1 -- 172.21.15.28:0/3878470992 >> v1:172.21.15.28:6789/0 conn(0x7fae540c6720 legacy=0x7fae540c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.029+0000 7fae59200640 1 -- 172.21.15.28:0/3878470992 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5415a1e0 con 0x7fae54156c30 2024-09-13T16:48:08.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.029+0000 7fae53400640 1 -- 172.21.15.28:0/3878470992 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453372682 0 0) 0x7fae54159f70 con 0x7fae540c6720 2024-09-13T16:48:08.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.030+0000 7fae59200640 1 --2- 172.21.15.28:0/3878470992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae54156c30 0x7fae54157040 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fae5406c9f0 tx=0x7fae4c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a0626f72bd3338 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.030+0000 7fae53400640 1 -- 172.21.15.28:0/3878470992 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c007740 con 0x7fae54156c30 2024-09-13T16:48:08.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.030+0000 7fae53400640 1 -- 172.21.15.28:0/3878470992 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4c02d050 con 0x7fae54156c30 2024-09-13T16:48:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.030+0000 7fae53400640 1 -- 172.21.15.28:0/3878470992 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c007ac0 con 0x7fae54156c30 2024-09-13T16:48:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.030+0000 7fae5b200640 1 -- 172.21.15.28:0/3878470992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae54156c30 msgr2=0x7fae54157040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.030+0000 7fae5b200640 1 --2- 172.21.15.28:0/3878470992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae54156c30 0x7fae54157040 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fae5406c9f0 tx=0x7fae4c02a710 comp rx=0 tx=0).stop 2024-09-13T16:48:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.031+0000 7fae5b200640 1 -- 172.21.15.28:0/3878470992 shutdown_connections 2024-09-13T16:48:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.031+0000 7fae5b200640 1 --2- 172.21.15.28:0/3878470992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae54156c30 0x7fae54157040 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.031+0000 7fae5b200640 1 --2- 172.21.15.28:0/3878470992 >> v2:172.21.15.191:3300/0 conn(0x7fae54150a70 0x7fae540c1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.031+0000 7fae5b200640 1 -- 172.21.15.28:0/3878470992 >> 172.21.15.28:0/3878470992 conn(0x7fae54069e80 msgr2=0x7fae5406a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.031+0000 7fae5b200640 1 -- 172.21.15.28:0/3878470992 shutdown_connections 2024-09-13T16:48:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.031+0000 7fae5b200640 1 -- 172.21.15.28:0/3878470992 wait complete. 2024-09-13T16:48:08.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.032+0000 7fae5b200640 1 Processor -- start 2024-09-13T16:48:08.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.032+0000 7fae5b200640 1 -- start start 2024-09-13T16:48:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.033+0000 7fae5b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae540c6720 0x7fae54147820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.033+0000 7fae5b200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae54150a70 0x7fae54158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.033+0000 7fae5b200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae54156c30 0x7fae54070d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.033+0000 7fae5b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae54071c60 con 0x7fae540c6720 2024-09-13T16:48:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.033+0000 7fae5b200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae54070900 con 0x7fae54150a70 2024-09-13T16:48:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.033+0000 7fae5b200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae54071e90 con 0x7fae54156c30 2024-09-13T16:48:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.033+0000 7fae53e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae540c6720 0x7fae54147820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.033+0000 7fae53e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae540c6720 0x7fae54147820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60210/0 (socket says 172.21.15.28:60210) 2024-09-13T16:48:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.034+0000 7fae59200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae54156c30 0x7fae54070d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.034+0000 7fae53e00640 1 -- 172.21.15.28:0/2561462153 learned_addr learned my addr 172.21.15.28:0/2561462153 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.034+0000 7fae4be00640 1 --2- 172.21.15.28:0/2561462153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae54150a70 0x7fae54158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.034+0000 7fae53e00640 1 -- 172.21.15.28:0/2561462153 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae54156c30 msgr2=0x7fae54070d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.034+0000 7fae53e00640 1 --2- 172.21.15.28:0/2561462153 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae54156c30 0x7fae54070d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.034+0000 7fae53e00640 1 -- 172.21.15.28:0/2561462153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae54150a70 msgr2=0x7fae54158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.034+0000 7fae53e00640 1 --2- 172.21.15.28:0/2561462153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae54150a70 0x7fae54158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.034+0000 7fae53e00640 1 -- 172.21.15.28:0/2561462153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c009a70 con 0x7fae540c6720 2024-09-13T16:48:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.034+0000 7fae53e00640 1 --2- 172.21.15.28:0/2561462153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae540c6720 0x7fae54147820 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fae4400e9b0 tx=0x7fae4400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.034+0000 7fae50c00640 1 -- 172.21.15.28:0/2561462153 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae44009670 con 0x7fae540c6720 2024-09-13T16:48:08.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.034+0000 7fae50c00640 1 -- 172.21.15.28:0/2561462153 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4400c9a0 con 0x7fae540c6720 2024-09-13T16:48:08.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.034+0000 7fae50c00640 1 -- 172.21.15.28:0/2561462153 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4400cd50 con 0x7fae540c6720 2024-09-13T16:48:08.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.035+0000 7fae5b200640 1 -- 172.21.15.28:0/2561462153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae541443c0 con 0x7fae540c6720 2024-09-13T16:48:08.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.035+0000 7fae5b200640 1 -- 172.21.15.28:0/2561462153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae54144c60 con 0x7fae540c6720 2024-09-13T16:48:08.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.036+0000 7fae5b200640 1 -- 172.21.15.28:0/2561462153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae540c4750 con 0x7fae540c6720 2024-09-13T16:48:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.037+0000 7fae50c00640 1 -- 172.21.15.28:0/2561462153 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fae44013470 con 0x7fae540c6720 2024-09-13T16:48:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.038+0000 7fae50c00640 1 --2- 172.21.15.28:0/2561462153 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fae3006ac00 0x7fae3006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.038+0000 7fae50c00640 1 -- 172.21.15.28:0/2561462153 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fae44089cf0 con 0x7fae540c6720 2024-09-13T16:48:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.038+0000 7fae4be00640 1 --2- 172.21.15.28:0/2561462153 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fae3006ac00 0x7fae3006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.039+0000 7fae4be00640 1 --2- 172.21.15.28:0/2561462153 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fae3006ac00 0x7fae3006d0c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fae5414df10 tx=0x7fae3c007d50 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.042+0000 7fae50c00640 1 -- 172.21.15.28:0/2561462153 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fae44054f70 con 0x7fae540c6720 2024-09-13T16:48:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.284+0000 7fae5b200640 1 -- 172.21.15.28:0/2561462153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fae540c5930 con 0x7fae540c6720 2024-09-13T16:48:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.284+0000 7fae50c00640 1 -- 172.21.15.28:0/2561462153 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1547 (secure 0 0 0) 0x7fae44021620 con 0x7fae540c6720 2024-09-13T16:48:08.287 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:48:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.286+0000 7fae5b200640 1 -- 172.21.15.28:0/2561462153 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fae3006ac00 msgr2=0x7fae3006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.286+0000 7fae5b200640 1 --2- 172.21.15.28:0/2561462153 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fae3006ac00 0x7fae3006d0c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fae5414df10 tx=0x7fae3c007d50 comp rx=0 tx=0).stop 2024-09-13T16:48:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.286+0000 7fae5b200640 1 -- 172.21.15.28:0/2561462153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae540c6720 msgr2=0x7fae54147820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.286+0000 7fae5b200640 1 --2- 172.21.15.28:0/2561462153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae540c6720 0x7fae54147820 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fae4400e9b0 tx=0x7fae4400ee70 comp rx=0 tx=0).stop 2024-09-13T16:48:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.286+0000 7fae5b200640 1 -- 172.21.15.28:0/2561462153 shutdown_connections 2024-09-13T16:48:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.286+0000 7fae5b200640 1 --2- 172.21.15.28:0/2561462153 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fae3006ac00 0x7fae3006d0c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.286+0000 7fae5b200640 1 --2- 172.21.15.28:0/2561462153 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fae54156c30 0x7fae54070d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.286+0000 7fae5b200640 1 --2- 172.21.15.28:0/2561462153 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fae54150a70 0x7fae54158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.286+0000 7fae5b200640 1 --2- 172.21.15.28:0/2561462153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae540c6720 0x7fae54147820 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.286+0000 7fae5b200640 1 -- 172.21.15.28:0/2561462153 >> 172.21.15.28:0/2561462153 conn(0x7fae54069e80 msgr2=0x7fae540c5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.287+0000 7fae5b200640 1 -- 172.21.15.28:0/2561462153 shutdown_connections 2024-09-13T16:48:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.287+0000 7fae5b200640 1 -- 172.21.15.28:0/2561462153 wait complete. 2024-09-13T16:48:08.306 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T16:48:08.306 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-13T16:48:08.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T16:48:08.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.460+0000 7f8919e00640 1 Processor -- start 2024-09-13T16:48:08.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.461+0000 7f8919e00640 1 -- start start 2024-09-13T16:48:08.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.462+0000 7f8919e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f891406ffa0 0x7f891406cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.462+0000 7f8919e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f891406e9b0 con 0x7f891415c230 2024-09-13T16:48:08.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.462+0000 7f8919e00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89140c3440 con 0x7f891415cc60 2024-09-13T16:48:08.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.462+0000 7f8919e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89140c3260 con 0x7f891406ffa0 2024-09-13T16:48:08.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.462+0000 7f8913e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f891415c230 0x7f891406f4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50548/0 (socket says 172.21.15.28:50548) 2024-09-13T16:48:08.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.462+0000 7f8913e00640 1 -- 172.21.15.28:0/1685991415 learned_addr learned my addr 172.21.15.28:0/1685991415 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:08.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.462+0000 7f8919400640 1 --2- 172.21.15.28:0/1685991415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f891406ffa0 0x7f891406cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.463+0000 7f8913400640 1 -- 172.21.15.28:0/1685991415 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461376550 0 0) 0x7f89140c3440 con 0x7f891415cc60 2024-09-13T16:48:08.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.463+0000 7f8919400640 1 -- 172.21.15.28:0/1685991415 >> v1:172.21.15.191:6789/0 conn(0x7f891415cc60 legacy=0x7f891406b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.463+0000 7f8919400640 1 -- 172.21.15.28:0/1685991415 >> v1:172.21.15.28:6789/0 conn(0x7f891415c230 legacy=0x7f891406f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.463+0000 7f8919400640 1 -- 172.21.15.28:0/1685991415 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89140c3720 con 0x7f891406ffa0 2024-09-13T16:48:08.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.464+0000 7f8919400640 1 --2- 172.21.15.28:0/1685991415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f891406ffa0 0x7f891406cd30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f890c009dc0 tx=0x7f890c02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82362a9fdb7a41b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.464+0000 7f8913400640 1 -- 172.21.15.28:0/1685991415 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c007740 con 0x7f891406ffa0 2024-09-13T16:48:08.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.464+0000 7f8913400640 1 -- 172.21.15.28:0/1685991415 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890c02d050 con 0x7f891406ffa0 2024-09-13T16:48:08.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.464+0000 7f8913400640 1 -- 172.21.15.28:0/1685991415 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c007ac0 con 0x7f891406ffa0 2024-09-13T16:48:08.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.464+0000 7f8919e00640 1 -- 172.21.15.28:0/1685991415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f891406ffa0 msgr2=0x7f891406cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.464+0000 7f8919e00640 1 --2- 172.21.15.28:0/1685991415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f891406ffa0 0x7f891406cd30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f890c009dc0 tx=0x7f890c02a660 comp rx=0 tx=0).stop 2024-09-13T16:48:08.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.464+0000 7f8919e00640 1 -- 172.21.15.28:0/1685991415 shutdown_connections 2024-09-13T16:48:08.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.464+0000 7f8919e00640 1 --2- 172.21.15.28:0/1685991415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f891406ffa0 0x7f891406cd30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.464+0000 7f8919e00640 1 -- 172.21.15.28:0/1685991415 >> 172.21.15.28:0/1685991415 conn(0x7f89140cc6a0 msgr2=0x7f89140ccab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:08.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.465+0000 7f8919e00640 1 -- 172.21.15.28:0/1685991415 shutdown_connections 2024-09-13T16:48:08.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.465+0000 7f8919e00640 1 -- 172.21.15.28:0/1685991415 wait complete. 2024-09-13T16:48:08.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.466+0000 7f8919e00640 1 Processor -- start 2024-09-13T16:48:08.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.466+0000 7f8919e00640 1 -- start start 2024-09-13T16:48:08.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.467+0000 7f8919e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f891406ffa0 0x7f89140b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.467+0000 7f8919e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f891415c230 0x7f89141553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.467+0000 7f8919e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f891415cc60 0x7f89140afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.467+0000 7f8919e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8914159e20 con 0x7f891415c230 2024-09-13T16:48:08.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.467+0000 7f8919e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89140af7c0 con 0x7f891415cc60 2024-09-13T16:48:08.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.467+0000 7f8919e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89141ee0c0 con 0x7f891406ffa0 2024-09-13T16:48:08.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.467+0000 7f8913e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f891415c230 0x7f89141553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.467+0000 7f8913e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f891415c230 0x7f89141553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60216/0 (socket says 172.21.15.28:60216) 2024-09-13T16:48:08.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.467+0000 7f8913e00640 1 -- 172.21.15.28:0/1178296527 learned_addr learned my addr 172.21.15.28:0/1178296527 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:08.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.467+0000 7f8919400640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f891415cc60 0x7f89140afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.468+0000 7f8918a00640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f891406ffa0 0x7f89140b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.468+0000 7f8913e00640 1 -- 172.21.15.28:0/1178296527 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f891406ffa0 msgr2=0x7f89140b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.468+0000 7f8913e00640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f891406ffa0 0x7f89140b1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.468+0000 7f8913e00640 1 -- 172.21.15.28:0/1178296527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f891415cc60 msgr2=0x7f89140afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.468+0000 7f8913e00640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f891415cc60 0x7f89140afc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.468+0000 7f8913e00640 1 -- 172.21.15.28:0/1178296527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890c009a70 con 0x7f891415c230 2024-09-13T16:48:08.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.468+0000 7f8919400640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f891415cc60 0x7f89140afc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:48:08.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.468+0000 7f8913e00640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f891415c230 0x7f89141553d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f890000c780 tx=0x7f890000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.468+0000 7f8910c00640 1 -- 172.21.15.28:0/1178296527 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89000072d0 con 0x7f891415c230 2024-09-13T16:48:08.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.468+0000 7f8910c00640 1 -- 172.21.15.28:0/1178296527 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8900007450 con 0x7f891415c230 2024-09-13T16:48:08.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.468+0000 7f8919e00640 1 -- 172.21.15.28:0/1178296527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f891415b090 con 0x7f891415c230 2024-09-13T16:48:08.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.468+0000 7f8910c00640 1 -- 172.21.15.28:0/1178296527 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89000047e0 con 0x7f891415c230 2024-09-13T16:48:08.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.469+0000 7f8919e00640 1 -- 172.21.15.28:0/1178296527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f891415a350 con 0x7f891415c230 2024-09-13T16:48:08.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.469+0000 7f8919e00640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89140b0190 0x7f89141ee610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.469+0000 7f8919e00640 1 -- 172.21.15.28:0/1178296527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891406ab50 con 0x7f89140b0190 2024-09-13T16:48:08.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.470+0000 7f8918a00640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89140b0190 0x7f89141ee610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.470+0000 7f8918a00640 1 -- 172.21.15.28:0/1178296527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f891415d040 con 0x7f89140b0190 2024-09-13T16:48:08.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.470+0000 7f8918a00640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89140b0190 0x7f89141ee610 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8908000f60 tx=0x7f890800e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.471+0000 7f8910c00640 1 -- 172.21.15.28:0/1178296527 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89080052e0 con 0x7f89140b0190 2024-09-13T16:48:08.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.471+0000 7f8910c00640 1 -- 172.21.15.28:0/1178296527 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8900004960 con 0x7f891415c230 2024-09-13T16:48:08.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.472+0000 7f8910c00640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f88f406aca0 0x7f88f406d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.472+0000 7f8910c00640 1 -- 172.21.15.28:0/1178296527 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8900088aa0 con 0x7f891415c230 2024-09-13T16:48:08.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.472+0000 7f8910c00640 1 -- 172.21.15.28:0/1178296527 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f8908005440 con 0x7f89140b0190 2024-09-13T16:48:08.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.472+0000 7f8910c00640 1 -- 172.21.15.28:0/1178296527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89140b0190 msgr2=0x7f89141ee610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.472+0000 7f8910c00640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89140b0190 0x7f89141ee610 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8908000f60 tx=0x7f890800e960 comp rx=0 tx=0).stop 2024-09-13T16:48:08.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.472+0000 7f8910c00640 1 -- 172.21.15.28:0/1178296527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89140b0190 msgr2=0x7f89141ee610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:48:08.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.472+0000 7f8910c00640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89140b0190 0x7f89141ee610 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8908000f60 tx=0x7f890800e960 comp rx=0 tx=0).stop 2024-09-13T16:48:08.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.472+0000 7f8919400640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f88f406aca0 0x7f88f406d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.473+0000 7f8919400640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f88f406aca0 0x7f88f406d160 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f890c034000 tx=0x7f890c02f000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.479+0000 7f8919e00640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89140c9860 0x7f89141eeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.479+0000 7f8919e00640 1 -- 172.21.15.28:0/1178296527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891406eee0 con 0x7f89140c9860 2024-09-13T16:48:08.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.479+0000 7f8918a00640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89140c9860 0x7f89141eeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.479+0000 7f8918a00640 1 -- 172.21.15.28:0/1178296527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f891406fbb0 con 0x7f89140c9860 2024-09-13T16:48:08.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.480+0000 7f8918a00640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89140c9860 0x7f89141eeb50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8908000f60 tx=0x7f8908001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.480+0000 7f8910c00640 1 -- 172.21.15.28:0/1178296527 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908003b70 con 0x7f89140c9860 2024-09-13T16:48:08.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.480+0000 7f8910c00640 1 -- 172.21.15.28:0/1178296527 <== mon.0 v2:172.21.15.28:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4459 (secure 0 0 0) 0x7f8908003cd0 con 0x7f89140c9860 2024-09-13T16:48:08.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.480+0000 7f8910c00640 1 -- 172.21.15.28:0/1178296527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89140c9860 msgr2=0x7f89141eeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.480+0000 7f8910c00640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89140c9860 0x7f89141eeb50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8908000f60 tx=0x7f8908001080 comp rx=0 tx=0).stop 2024-09-13T16:48:08.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.480+0000 7f8910c00640 1 -- 172.21.15.28:0/1178296527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89140c9860 msgr2=0x7f89141eeb50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:48:08.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.480+0000 7f8910c00640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89140c9860 0x7f89141eeb50 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8908000f60 tx=0x7f8908001080 comp rx=0 tx=0).stop 2024-09-13T16:48:08.487 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:48:08.487 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:48:08.487 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:48:08.487 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "leader", 2024-09-13T16:48:08.488 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 30, 2024-09-13T16:48:08.488 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:48:08.488 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:48:08.488 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:48:08.488 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:48:08.488 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:48:08.489 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 1, 2024-09-13T16:48:08.489 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:48:08.489 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:48:08.489 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:48:08.489 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:48:08.489 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:48:08.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:48:08.490 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:48:08.490 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:48:08.490 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:48:08.490 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:48:08.490 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:48:08.490 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:48:08.491 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:48:08.491 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:48:08.491 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:48:08.491 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:48:08.491 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:48:08.491 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:48:08.491 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:48:08.492 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:48:08.492 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:48:08.492 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:48:08.492 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:48:08.492 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:48:08.492 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:48:08.493 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:48:08.493 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.493 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:48:08.493 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:48:08.493 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:48:08.493 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:48:08.494 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:48:08.494 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:48:08.494 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:48:08.494 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:48:08.494 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:48:08.494 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:48:08.495 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:48:08.495 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:48:08.495 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:48:08.495 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:48:08.495 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:48:08.495 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:48:08.495 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:48:08.496 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:48:08.496 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:48:08.496 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:48:08.496 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:48:08.496 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:48:08.496 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:48:08.497 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:48:08.497 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:48:08.497 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:48:08.497 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:48:08.497 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:48:08.497 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:48:08.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.498 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:48:08.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.498 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:48:08.498 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:48:08.498 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:48:08.498 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:48:08.499 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.499 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:48:08.499 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:48:08.499 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.500 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:48:08.500 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:48:08.500 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.500 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.500 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.501 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:48:08.501 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:48:08.501 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:48:08.501 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:48:08.501 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:48:08.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.502 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:48:08.502 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:48:08.502 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:48:08.502 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:48:08.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.503 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:48:08.503 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:48:08.503 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.504 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:48:08.504 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:48:08.504 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.504 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.504 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.505 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:48:08.505 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:48:08.505 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:48:08.505 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:48:08.505 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:48:08.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.506 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:48:08.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:48:08.506 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:48:08.506 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:48:08.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.506 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:48:08.507 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:48:08.507 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.507 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.507 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:48:08.507 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:48:08.507 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.508 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.508 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.508 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:48:08.508 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:48:08.508 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:48:08.508 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:48:08.509 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:48:08.509 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.509 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.509 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:48:08.509 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:48:08.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.510 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:48:08.510 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:48:08.510 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:48:08.510 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.510 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:48:08.510 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:48:08.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.511 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:48:08.511 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:48:08.511 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:48:08.511 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.511 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.512 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:48:08.512 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:48:08.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.481+0000 7f8919e00640 1 -- 172.21.15.28:0/1178296527 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f88f406aca0 msgr2=0x7f88f406d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.481+0000 7f8919e00640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f88f406aca0 0x7f88f406d160 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f890c034000 tx=0x7f890c02f000 comp rx=0 tx=0).stop 2024-09-13T16:48:08.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.481+0000 7f8919e00640 1 -- 172.21.15.28:0/1178296527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f891415c230 msgr2=0x7f89141553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.481+0000 7f8919e00640 1 --2- 172.21.15.28:0/1178296527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f891415c230 0x7f89141553d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f890000c780 tx=0x7f890000cc40 comp rx=0 tx=0).stop 2024-09-13T16:48:08.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.481+0000 7f8919400640 1 -- 172.21.15.28:0/1178296527 reap_dead start 2024-09-13T16:48:08.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.481+0000 7f8919400640 1 -- 172.21.15.28:0/1178296527 reap_dead start 2024-09-13T16:48:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.481+0000 7f8919e00640 1 -- 172.21.15.28:0/1178296527 shutdown_connections 2024-09-13T16:48:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.481+0000 7f8919e00640 1 -- 172.21.15.28:0/1178296527 >> 172.21.15.28:0/1178296527 conn(0x7f89140cc6a0 msgr2=0x7f89140c2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.481+0000 7f8919e00640 1 -- 172.21.15.28:0/1178296527 shutdown_connections 2024-09-13T16:48:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.481+0000 7f8919e00640 1 -- 172.21.15.28:0/1178296527 wait complete. 2024-09-13T16:48:08.517 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:48:08.517 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-13T16:48:08.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T16:48:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.672+0000 7f4a17200640 1 Processor -- start 2024-09-13T16:48:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.672+0000 7f4a17200640 1 -- start start 2024-09-13T16:48:08.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.673+0000 7f4a17200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4a180ce9c0 0x7f4a181595a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.673+0000 7f4a17200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4a180c94d0 0x7f4a180c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.673+0000 7f4a17200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a18159f90 con 0x7f4a180ce5e0 2024-09-13T16:48:08.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.673+0000 7f4a17200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a18159ae0 con 0x7f4a180c94d0 2024-09-13T16:48:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.673+0000 7f4a17200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a180c6900 con 0x7f4a180ce9c0 2024-09-13T16:48:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.673+0000 7f4a16800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4a180c94d0 0x7f4a180c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.673+0000 7f4a15e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4a180ce5e0 0x7f4a1815b590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50560/0 (socket says 172.21.15.28:50560) 2024-09-13T16:48:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.673+0000 7f4a15e00640 1 -- 172.21.15.28:0/461073136 learned_addr learned my addr 172.21.15.28:0/461073136 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.673+0000 7f4a15400640 1 --2- 172.21.15.28:0/461073136 >> v2:172.21.15.191:3300/0 conn(0x7f4a180ce9c0 0x7f4a181595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.674+0000 7f4a16800640 1 -- 172.21.15.28:0/461073136 >> v2:172.21.15.191:3300/0 conn(0x7f4a180ce9c0 msgr2=0x7f4a181595a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.674+0000 7f4a16800640 1 --2- 172.21.15.28:0/461073136 >> v2:172.21.15.191:3300/0 conn(0x7f4a180ce9c0 0x7f4a181595a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.674+0000 7f4a16800640 1 -- 172.21.15.28:0/461073136 >> v1:172.21.15.28:6789/0 conn(0x7f4a180ce5e0 legacy=0x7f4a1815b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.674+0000 7f4a16800640 1 -- 172.21.15.28:0/461073136 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a180c5800 con 0x7f4a180c94d0 2024-09-13T16:48:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.674+0000 7f4a14a00640 1 -- 172.21.15.28:0/461073136 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766062238 0 0) 0x7f4a18159f90 con 0x7f4a180ce5e0 2024-09-13T16:48:08.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.675+0000 7f4a16800640 1 --2- 172.21.15.28:0/461073136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a180c94d0 0x7f4a180c6d90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4a180c4fe0 tx=0x7f4a1002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ac56c9223e6eb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.675+0000 7f4a14a00640 1 -- 172.21.15.28:0/461073136 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a10007740 con 0x7f4a180c94d0 2024-09-13T16:48:08.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.675+0000 7f4a14a00640 1 -- 172.21.15.28:0/461073136 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1002d050 con 0x7f4a180c94d0 2024-09-13T16:48:08.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.675+0000 7f4a14a00640 1 -- 172.21.15.28:0/461073136 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a10007ac0 con 0x7f4a180c94d0 2024-09-13T16:48:08.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.676+0000 7f4a17200640 1 -- 172.21.15.28:0/461073136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a180c94d0 msgr2=0x7f4a180c6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.676+0000 7f4a17200640 1 --2- 172.21.15.28:0/461073136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a180c94d0 0x7f4a180c6d90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4a180c4fe0 tx=0x7f4a1002a710 comp rx=0 tx=0).stop 2024-09-13T16:48:08.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.676+0000 7f4a17200640 1 -- 172.21.15.28:0/461073136 shutdown_connections 2024-09-13T16:48:08.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.676+0000 7f4a17200640 1 --2- 172.21.15.28:0/461073136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a180c94d0 0x7f4a180c6d90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.676+0000 7f4a17200640 1 --2- 172.21.15.28:0/461073136 >> v2:172.21.15.191:3300/0 conn(0x7f4a180ce9c0 0x7f4a181595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.676+0000 7f4a17200640 1 -- 172.21.15.28:0/461073136 >> 172.21.15.28:0/461073136 conn(0x7f4a1806c880 msgr2=0x7f4a180715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:08.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.676+0000 7f4a17200640 1 -- 172.21.15.28:0/461073136 shutdown_connections 2024-09-13T16:48:08.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.676+0000 7f4a17200640 1 -- 172.21.15.28:0/461073136 wait complete. 2024-09-13T16:48:08.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.677+0000 7f4a17200640 1 Processor -- start 2024-09-13T16:48:08.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.677+0000 7f4a17200640 1 -- start start 2024-09-13T16:48:08.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.678+0000 7f4a17200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a180c94d0 0x7f4a1814faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.678+0000 7f4a17200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a180ce5e0 0x7f4a18157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.678+0000 7f4a17200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a180ce9c0 0x7f4a18151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.678+0000 7f4a17200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a18158e50 con 0x7f4a180ce5e0 2024-09-13T16:48:08.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.678+0000 7f4a17200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a181eebb0 con 0x7f4a180c94d0 2024-09-13T16:48:08.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.678+0000 7f4a17200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a180caa70 con 0x7f4a180ce9c0 2024-09-13T16:48:08.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.678+0000 7f4a15400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a180ce5e0 0x7f4a18157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.679+0000 7f4a15400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a180ce5e0 0x7f4a18157ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60234/0 (socket says 172.21.15.28:60234) 2024-09-13T16:48:08.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.679+0000 7f4a15400640 1 -- 172.21.15.28:0/3660874996 learned_addr learned my addr 172.21.15.28:0/3660874996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:08.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.679+0000 7f4a15e00640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a180c94d0 0x7f4a1814faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.679+0000 7f4a16800640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a180ce9c0 0x7f4a18151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.679+0000 7f4a15400640 1 -- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a180ce9c0 msgr2=0x7f4a18151720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.679+0000 7f4a15400640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4a180ce9c0 0x7f4a18151720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.679+0000 7f4a15400640 1 -- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a180c94d0 msgr2=0x7f4a1814faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.679+0000 7f4a15400640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a180c94d0 0x7f4a1814faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.679+0000 7f4a15400640 1 -- 172.21.15.28:0/3660874996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a10009a70 con 0x7f4a180ce5e0 2024-09-13T16:48:08.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.679+0000 7f4a15400640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a180ce5e0 0x7f4a18157ae0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400e870 tx=0x7f4a0400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.679+0000 7f4a0e000640 1 -- 172.21.15.28:0/3660874996 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a040094f0 con 0x7f4a180ce5e0 2024-09-13T16:48:08.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.679+0000 7f4a0e000640 1 -- 172.21.15.28:0/3660874996 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0400c940 con 0x7f4a180ce5e0 2024-09-13T16:48:08.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.679+0000 7f4a0e000640 1 -- 172.21.15.28:0/3660874996 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0400ccf0 con 0x7f4a180ce5e0 2024-09-13T16:48:08.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.680+0000 7f4a17200640 1 -- 172.21.15.28:0/3660874996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a180cad50 con 0x7f4a180ce5e0 2024-09-13T16:48:08.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.680+0000 7f4a17200640 1 -- 172.21.15.28:0/3660874996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a181575c0 con 0x7f4a180ce5e0 2024-09-13T16:48:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.681+0000 7f4a17200640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a18159700 0x7f4a18151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.681+0000 7f4a17200640 1 -- 172.21.15.28:0/3660874996 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a18072160 con 0x7f4a18159700 2024-09-13T16:48:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.681+0000 7f4a15e00640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a18159700 0x7f4a18151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.681+0000 7f4a15e00640 1 -- 172.21.15.28:0/3660874996 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a180c5cf0 con 0x7f4a18159700 2024-09-13T16:48:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.682+0000 7f4a15e00640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a18159700 0x7f4a18151c60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4a080011e0 tx=0x7f4a08001990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.682+0000 7f4a0e000640 1 -- 172.21.15.28:0/3660874996 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4a04013470 con 0x7f4a180ce5e0 2024-09-13T16:48:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.683+0000 7f4a0e000640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f49f806ac30 0x7f49f806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.683+0000 7f4a0e000640 1 -- 172.21.15.28:0/3660874996 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4a04089410 con 0x7f4a180ce5e0 2024-09-13T16:48:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.683+0000 7f4a0e000640 1 -- 172.21.15.28:0/3660874996 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0800f600 con 0x7f4a18159700 2024-09-13T16:48:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.683+0000 7f4a0e000640 1 -- 172.21.15.28:0/3660874996 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f4a0800f760 con 0x7f4a18159700 2024-09-13T16:48:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.683+0000 7f4a0e000640 1 -- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a18159700 msgr2=0x7f4a18151c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.683+0000 7f4a0e000640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a18159700 0x7f4a18151c60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4a080011e0 tx=0x7f4a08001990 comp rx=0 tx=0).stop 2024-09-13T16:48:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.683+0000 7f4a16800640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f49f806ac30 0x7f49f806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.683+0000 7f4a0e000640 1 -- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a18159700 msgr2=0x7f4a18151c60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:48:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.683+0000 7f4a0e000640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a18159700 0x7f4a18151c60 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4a080011e0 tx=0x7f4a08001990 comp rx=0 tx=0).stop 2024-09-13T16:48:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.683+0000 7f4a16800640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f49f806ac30 0x7f49f806d0f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4a10034000 tx=0x7f4a1002f000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.690+0000 7f4a17200640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a181ee670 0x7f4a181f1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.690+0000 7f4a17200640 1 -- 172.21.15.28:0/3660874996 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a181577b0 con 0x7f4a181ee670 2024-09-13T16:48:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.690+0000 7f4a15e00640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a181ee670 0x7f4a181f1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.690+0000 7f4a15e00640 1 -- 172.21.15.28:0/3660874996 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4a180c8260 con 0x7f4a181ee670 2024-09-13T16:48:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.691+0000 7f4a15e00640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a181ee670 0x7f4a181f1160 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4a080011e0 tx=0x7f4a08001080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.691+0000 7f4a0e000640 1 -- 172.21.15.28:0/3660874996 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a08022070 con 0x7f4a181ee670 2024-09-13T16:48:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.691+0000 7f4a0e000640 1 -- 172.21.15.28:0/3660874996 <== mon.1 v2:172.21.15.191:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4789 (secure 0 0 0) 0x7f4a08013600 con 0x7f4a181ee670 2024-09-13T16:48:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.691+0000 7f4a0e000640 1 -- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a181ee670 msgr2=0x7f4a181f1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.691+0000 7f4a0e000640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a181ee670 0x7f4a181f1160 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4a080011e0 tx=0x7f4a08001080 comp rx=0 tx=0).stop 2024-09-13T16:48:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.691+0000 7f4a0e000640 1 -- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a181ee670 msgr2=0x7f4a181f1160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:48:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.691+0000 7f4a0e000640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4a181ee670 0x7f4a181f1160 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4a080011e0 tx=0x7f4a08001080 comp rx=0 tx=0).stop 2024-09-13T16:48:08.695 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:48:08.695 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:48:08.695 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:48:08.695 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:48:08.695 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 30, 2024-09-13T16:48:08.695 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:48:08.695 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:48:08.695 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:48:08.695 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:48:08.695 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:48:08.695 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 1, 2024-09-13T16:48:08.695 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:48:08.696 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:48:08.696 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:48:08.696 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:48:08.696 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:48:08.696 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:48:08.696 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:48:08.696 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:48:08.696 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:48:08.696 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:48:08.696 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:48:08.696 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:48:08.696 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:48:08.696 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:48:08.697 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:48:08.697 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:48:08.697 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:48:08.697 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:48:08.697 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:48:08.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:48:08.697 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:48:08.697 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:48:08.697 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:48:08.697 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:48:08.697 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:48:08.697 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:48:08.697 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.698 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:48:08.698 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:48:08.698 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:48:08.698 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:48:08.698 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:48:08.698 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:48:08.698 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:48:08.698 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:48:08.698 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:48:08.698 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:48:08.698 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:48:08.698 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:48:08.698 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:48:08.699 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:48:08.699 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:48:08.699 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:48:08.699 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:48:08.699 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:48:08.699 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:48:08.699 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:48:08.699 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:48:08.699 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:48:08.699 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:48:08.699 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:48:08.699 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:48:08.699 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:48:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:48:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:48:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:48:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:48:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:48:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:48:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:48:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:48:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:48:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:48:08.701 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.701 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:48:08.701 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:48:08.701 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.701 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.701 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.701 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:48:08.701 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:48:08.701 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:48:08.701 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:48:08.701 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:48:08.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.702 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:48:08.702 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:48:08.702 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:48:08.702 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:48:08.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.702 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:48:08.702 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:48:08.702 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.702 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:48:08.702 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:48:08.703 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.703 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.703 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:48:08.703 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:48:08.703 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:48:08.703 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:48:08.703 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:48:08.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.703 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:48:08.703 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:48:08.704 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:48:08.704 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:48:08.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:48:08.704 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:48:08.704 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:48:08.704 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:48:08.704 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.704 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.704 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.705 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:48:08.705 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:48:08.705 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:48:08.705 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:48:08.705 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:48:08.705 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.705 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.705 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:48:08.705 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:48:08.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.705 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:48:08.705 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:48:08.705 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:48:08.706 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.706 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:48:08.706 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:48:08.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.706 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:48:08.706 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:48:08.706 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:48:08.706 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.706 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:48:08.706 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:48:08.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.706 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:48:08.706 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:48:08.707 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:48:08.707 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.707 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:48:08.707 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:48:08.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.707 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:48:08.707 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:48:08.707 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:48:08.707 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.707 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.707 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:48:08.707 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:48:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.692+0000 7f4a17200640 1 -- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f49f806ac30 msgr2=0x7f49f806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.692+0000 7f4a17200640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f49f806ac30 0x7f49f806d0f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4a10034000 tx=0x7f4a1002f000 comp rx=0 tx=0).stop 2024-09-13T16:48:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.692+0000 7f4a17200640 1 -- 172.21.15.28:0/3660874996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a180ce5e0 msgr2=0x7f4a18157ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.692+0000 7f4a17200640 1 --2- 172.21.15.28:0/3660874996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a180ce5e0 0x7f4a18157ae0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400e870 tx=0x7f4a0400ed30 comp rx=0 tx=0).stop 2024-09-13T16:48:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.692+0000 7f4a16800640 1 -- 172.21.15.28:0/3660874996 reap_dead start 2024-09-13T16:48:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.692+0000 7f4a16800640 1 -- 172.21.15.28:0/3660874996 reap_dead start 2024-09-13T16:48:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.692+0000 7f4a17200640 1 -- 172.21.15.28:0/3660874996 shutdown_connections 2024-09-13T16:48:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.692+0000 7f4a17200640 1 -- 172.21.15.28:0/3660874996 >> 172.21.15.28:0/3660874996 conn(0x7f4a1806c880 msgr2=0x7f4a1815c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.692+0000 7f4a17200640 1 -- 172.21.15.28:0/3660874996 shutdown_connections 2024-09-13T16:48:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.692+0000 7f4a17200640 1 -- 172.21.15.28:0/3660874996 wait complete. 2024-09-13T16:48:08.710 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:48:08.710 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-13T16:48:08.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T16:48:08.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.864+0000 7f9beac00640 1 Processor -- start 2024-09-13T16:48:08.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.864+0000 7f9beac00640 1 -- start start 2024-09-13T16:48:08.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.866+0000 7f9beac00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9be415a440 0x7f9be40c5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.866+0000 7f9beac00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9be406dcb0 0x7f9be41504e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.866+0000 7f9beac00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be40c1d80 con 0x7f9be415a060 2024-09-13T16:48:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.866+0000 7f9beac00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be415a920 con 0x7f9be406dcb0 2024-09-13T16:48:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.866+0000 7f9beac00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be415ac00 con 0x7f9be415a440 2024-09-13T16:48:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.866+0000 7f9be8c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9be406dcb0 0x7f9be41504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.866+0000 7f9be8c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9be406dcb0 0x7f9be41504e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59550/0 (socket says 172.21.15.28:59550) 2024-09-13T16:48:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.866+0000 7f9be8c00640 1 -- 172.21.15.28:0/2046611337 learned_addr learned my addr 172.21.15.28:0/2046611337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.866+0000 7f9bdbe00640 1 --2- 172.21.15.28:0/2046611337 >> v2:172.21.15.191:3300/0 conn(0x7f9be415a440 0x7f9be40c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.866+0000 7f9be8c00640 1 -- 172.21.15.28:0/2046611337 >> v2:172.21.15.191:3300/0 conn(0x7f9be415a440 msgr2=0x7f9be40c5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.866+0000 7f9be8c00640 1 --2- 172.21.15.28:0/2046611337 >> v2:172.21.15.191:3300/0 conn(0x7f9be415a440 0x7f9be40c5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.866+0000 7f9be8c00640 1 -- 172.21.15.28:0/2046611337 >> v1:172.21.15.191:6789/0 conn(0x7f9be415a060 legacy=0x7f9be406d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.866+0000 7f9be8c00640 1 -- 172.21.15.28:0/2046611337 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be40c2d50 con 0x7f9be406dcb0 2024-09-13T16:48:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.867+0000 7f9be8c00640 1 --2- 172.21.15.28:0/2046611337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be406dcb0 0x7f9be41504e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9be40c47e0 tx=0x7f9bdc02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e2bc8cda0a13a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.868+0000 7f9be3400640 1 -- 172.21.15.28:0/2046611337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc031600 con 0x7f9be406dcb0 2024-09-13T16:48:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.868+0000 7f9be3400640 1 -- 172.21.15.28:0/2046611337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bdc02d050 con 0x7f9be406dcb0 2024-09-13T16:48:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.868+0000 7f9be3400640 1 -- 172.21.15.28:0/2046611337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc031980 con 0x7f9be406dcb0 2024-09-13T16:48:08.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.868+0000 7f9beac00640 1 -- 172.21.15.28:0/2046611337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be406dcb0 msgr2=0x7f9be41504e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.868+0000 7f9beac00640 1 --2- 172.21.15.28:0/2046611337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be406dcb0 0x7f9be41504e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9be40c47e0 tx=0x7f9bdc02a730 comp rx=0 tx=0).stop 2024-09-13T16:48:08.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.868+0000 7f9beac00640 1 -- 172.21.15.28:0/2046611337 shutdown_connections 2024-09-13T16:48:08.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.868+0000 7f9beac00640 1 --2- 172.21.15.28:0/2046611337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be406dcb0 0x7f9be41504e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.868+0000 7f9beac00640 1 --2- 172.21.15.28:0/2046611337 >> v2:172.21.15.191:3300/0 conn(0x7f9be415a440 0x7f9be40c5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.868+0000 7f9beac00640 1 -- 172.21.15.28:0/2046611337 >> 172.21.15.28:0/2046611337 conn(0x7f9be40ca6b0 msgr2=0x7f9be40caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:08.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.868+0000 7f9beac00640 1 -- 172.21.15.28:0/2046611337 shutdown_connections 2024-09-13T16:48:08.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.868+0000 7f9beac00640 1 -- 172.21.15.28:0/2046611337 wait complete. 2024-09-13T16:48:08.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.870+0000 7f9beac00640 1 Processor -- start 2024-09-13T16:48:08.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.870+0000 7f9beac00640 1 -- start start 2024-09-13T16:48:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.871+0000 7f9beac00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9be406dcb0 0x7f9be41efa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.871+0000 7f9beac00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be415a060 0x7f9be4157c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.871+0000 7f9beac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be415a440 0x7f9be4158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.871+0000 7f9beac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be414be20 con 0x7f9be415a440 2024-09-13T16:48:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.871+0000 7f9beac00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be41586b0 con 0x7f9be406dcb0 2024-09-13T16:48:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.871+0000 7f9beac00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be414e470 con 0x7f9be415a060 2024-09-13T16:48:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.871+0000 7f9be3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9be406dcb0 0x7f9be41efa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.871+0000 7f9bdbe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be415a060 0x7f9be4157c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.871+0000 7f9be3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9be406dcb0 0x7f9be41efa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:55204/0 (socket says 172.21.15.28:55204) 2024-09-13T16:48:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.871+0000 7f9bdbe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be415a060 0x7f9be4157c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:52092/0 (socket says 172.21.15.28:52092) 2024-09-13T16:48:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.871+0000 7f9be3e00640 1 -- 172.21.15.28:0/886170882 learned_addr learned my addr 172.21.15.28:0/886170882 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.871+0000 7f9be8c00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be415a440 0x7f9be4158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.872+0000 7f9bdbe00640 1 -- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9be406dcb0 msgr2=0x7f9be41efa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.872+0000 7f9bdbe00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9be406dcb0 0x7f9be41efa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.872+0000 7f9bdbe00640 1 -- 172.21.15.28:0/886170882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be415a440 msgr2=0x7f9be4158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.872+0000 7f9bdbe00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be415a440 0x7f9be4158170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.872+0000 7f9bdbe00640 1 -- 172.21.15.28:0/886170882 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bdc009b30 con 0x7f9be415a060 2024-09-13T16:48:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.872+0000 7f9bdbe00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be415a060 0x7f9be4157c30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9be40c5a40 tx=0x7f9bcc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.872+0000 7f9be0c00640 1 -- 172.21.15.28:0/886170882 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc00c8e0 con 0x7f9be415a060 2024-09-13T16:48:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.872+0000 7f9be0c00640 1 -- 172.21.15.28:0/886170882 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bcc00ca40 con 0x7f9be415a060 2024-09-13T16:48:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.872+0000 7f9be0c00640 1 -- 172.21.15.28:0/886170882 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc00cdd0 con 0x7f9be415a060 2024-09-13T16:48:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.872+0000 7f9beac00640 1 -- 172.21.15.28:0/886170882 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be41e6060 con 0x7f9be415a060 2024-09-13T16:48:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.873+0000 7f9beac00640 1 -- 172.21.15.28:0/886170882 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be4070900 con 0x7f9be415a060 2024-09-13T16:48:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.873+0000 7f9beac00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be40716f0 0x7f9be4158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.873+0000 7f9beac00640 1 -- 172.21.15.28:0/886170882 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be4066c20 con 0x7f9be40716f0 2024-09-13T16:48:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.874+0000 7f9be3e00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be40716f0 0x7f9be4158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.874+0000 7f9be3e00640 1 -- 172.21.15.28:0/886170882 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9be40c5220 con 0x7f9be40716f0 2024-09-13T16:48:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.874+0000 7f9be3e00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be40716f0 0x7f9be4158a70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400f770 tx=0x7f9bd400fc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.874+0000 7f9be0c00640 1 -- 172.21.15.28:0/886170882 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd40050d0 con 0x7f9be40716f0 2024-09-13T16:48:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.875+0000 7f9be0c00640 1 -- 172.21.15.28:0/886170882 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9bcc013470 con 0x7f9be415a060 2024-09-13T16:48:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.876+0000 7f9be0c00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f9bc006aca0 0x7f9bc006d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.876+0000 7f9be0c00640 1 -- 172.21.15.28:0/886170882 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f9bd4005230 con 0x7f9be40716f0 2024-09-13T16:48:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.876+0000 7f9be0c00640 1 -- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be40716f0 msgr2=0x7f9be4158a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.876+0000 7f9be0c00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be40716f0 0x7f9be4158a70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400f770 tx=0x7f9bd400fc30 comp rx=0 tx=0).stop 2024-09-13T16:48:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.876+0000 7f9be0c00640 1 -- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be40716f0 msgr2=0x7f9be4158a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:48:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.876+0000 7f9be0c00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be40716f0 0x7f9be4158a70 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400f770 tx=0x7f9bd400fc30 comp rx=0 tx=0).stop 2024-09-13T16:48:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.876+0000 7f9be0c00640 1 -- 172.21.15.28:0/886170882 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9bcc089780 con 0x7f9be415a060 2024-09-13T16:48:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.876+0000 7f9be8c00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f9bc006aca0 0x7f9bc006d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.877+0000 7f9be8c00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f9bc006aca0 0x7f9bc006d160 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc033000 tx=0x7f9bdc009000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.882+0000 7f9beac00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be4158fb0 0x7f9be41f1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.882+0000 7f9beac00640 1 -- 172.21.15.28:0/886170882 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be40c3310 con 0x7f9be4158fb0 2024-09-13T16:48:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.883+0000 7f9be3e00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be4158fb0 0x7f9be41f1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.883+0000 7f9be3e00640 1 -- 172.21.15.28:0/886170882 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9be40c19d0 con 0x7f9be4158fb0 2024-09-13T16:48:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.883+0000 7f9be3e00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be4158fb0 0x7f9be41f1b00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400f770 tx=0x7f9bd4001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.883+0000 7f9be0c00640 1 -- 172.21.15.28:0/886170882 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4023070 con 0x7f9be4158fb0 2024-09-13T16:48:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.884+0000 7f9be0c00640 1 -- 172.21.15.28:0/886170882 <== mon.2 v2:172.21.15.191:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+5098 (secure 0 0 0) 0x7f9bd400dc30 con 0x7f9be4158fb0 2024-09-13T16:48:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.884+0000 7f9be0c00640 1 -- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be4158fb0 msgr2=0x7f9be41f1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.884+0000 7f9be0c00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be4158fb0 0x7f9be41f1b00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400f770 tx=0x7f9bd4001120 comp rx=0 tx=0).stop 2024-09-13T16:48:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.884+0000 7f9be0c00640 1 -- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be4158fb0 msgr2=0x7f9be41f1b00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T16:48:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.884+0000 7f9be0c00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be4158fb0 0x7f9be41f1b00 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400f770 tx=0x7f9bd4001120 comp rx=0 tx=0).stop 2024-09-13T16:48:08.887 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:48:08.887 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:48:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:48:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "peon", 2024-09-13T16:48:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: "election_epoch": 30, 2024-09-13T16:48:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum": [ 2024-09-13T16:48:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-09-13T16:48:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-09-13T16:48:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-09-13T16:48:08.888 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:48:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_age": 1, 2024-09-13T16:48:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:48:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "required_con": "2449958758054445060", 2024-09-13T16:48:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "required_mon": [ 2024-09-13T16:48:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:48:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:48:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:48:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:48:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:48:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:48:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:48:08.889 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:48:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:48:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:48:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:48:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_con": "4540138322906710015", 2024-09-13T16:48:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: "quorum_mon": [ 2024-09-13T16:48:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:48:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:48:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:48:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:48:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:48:08.890 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:48:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:48:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:48:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:48:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:48:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "outside_quorum": [], 2024-09-13T16:48:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "extra_probe_peers": [], 2024-09-13T16:48:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "sync_provider": [], 2024-09-13T16:48:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": { 2024-09-13T16:48:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 1, 2024-09-13T16:48:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "8e702092-2b7f-4633-9ce6-17040007e53d", 2024-09-13T16:48:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:48:08.891 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-09-13T16:43:05.305835Z", 2024-09-13T16:48:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release": 18, 2024-09-13T16:48:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min_mon_release_name": "reef", 2024-09-13T16:48:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "election_strategy": 1, 2024-09-13T16:48:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "disallowed_leaders: ": "", 2024-09-13T16:48:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false, 2024-09-13T16:48:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "tiebreaker_mon": "", 2024-09-13T16:48:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_ranks: ": "", 2024-09-13T16:48:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "features": { 2024-09-13T16:48:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "persistent": [ 2024-09-13T16:48:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "kraken", 2024-09-13T16:48:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "luminous", 2024-09-13T16:48:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "mimic", 2024-09-13T16:48:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap-prune", 2024-09-13T16:48:08.892 INFO:teuthology.orchestra.run.smithi028.stdout: "nautilus", 2024-09-13T16:48:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus", 2024-09-13T16:48:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific", 2024-09-13T16:48:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "elector-pinging", 2024-09-13T16:48:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy", 2024-09-13T16:48:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "reef" 2024-09-13T16:48:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:48:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "optional": [] 2024-09-13T16:48:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "mons": [ 2024-09-13T16:48:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 0, 2024-09-13T16:48:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "a", 2024-09-13T16:48:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:48:08.893 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:48:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:48:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:3300", 2024-09-13T16:48:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:48:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789", 2024-09-13T16:48:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6789/0", 2024-09-13T16:48:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6789/0", 2024-09-13T16:48:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:48:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:48:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:48:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 1, 2024-09-13T16:48:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "b", 2024-09-13T16:48:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:48:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:48:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:48:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3300", 2024-09-13T16:48:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:48:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789", 2024-09-13T16:48:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6789/0", 2024-09-13T16:48:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6789/0", 2024-09-13T16:48:08.896 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:48:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:48:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:48:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: "rank": 2, 2024-09-13T16:48:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "c", 2024-09-13T16:48:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-09-13T16:48:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-09-13T16:48:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-09-13T16:48:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:3301", 2024-09-13T16:48:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-09-13T16:48:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790", 2024-09-13T16:48:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 0 2024-09-13T16:48:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.191:6790/0", 2024-09-13T16:48:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.191:6790/0", 2024-09-13T16:48:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "priority": 0, 2024-09-13T16:48:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-09-13T16:48:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "{}" 2024-09-13T16:48:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "feature_map": { 2024-09-13T16:48:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": [ 2024-09-13T16:48:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:48:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:48:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:48:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:48:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": [ 2024-09-13T16:48:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:48:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:48:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 5 2024-09-13T16:48:08.899 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:48:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": [ 2024-09-13T16:48:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:48:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:48:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 6 2024-09-13T16:48:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:48:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "client": [ 2024-09-13T16:48:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x2f018fb87aa4aafe", 2024-09-13T16:48:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "luminous", 2024-09-13T16:48:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 1 2024-09-13T16:48:08.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:48:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:48:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 5 2024-09-13T16:48:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T16:48:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": [ 2024-09-13T16:48:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T16:48:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "features": "0x3f01cfbffffdffff", 2024-09-13T16:48:08.901 INFO:teuthology.orchestra.run.smithi028.stdout: "release": "reef", 2024-09-13T16:48:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 2 2024-09-13T16:48:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T16:48:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T16:48:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T16:48:08.902 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": false 2024-09-13T16:48:08.902 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:48:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.884+0000 7f9beac00640 1 -- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f9bc006aca0 msgr2=0x7f9bc006d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.884+0000 7f9beac00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f9bc006aca0 0x7f9bc006d160 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc033000 tx=0x7f9bdc009000 comp rx=0 tx=0).stop 2024-09-13T16:48:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.884+0000 7f9beac00640 1 -- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be415a060 msgr2=0x7f9be4157c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.884+0000 7f9beac00640 1 --2- 172.21.15.28:0/886170882 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9be415a060 0x7f9be4157c30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9be40c5a40 tx=0x7f9bcc00ecb0 comp rx=0 tx=0).stop 2024-09-13T16:48:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.884+0000 7f9be8c00640 1 -- 172.21.15.28:0/886170882 reap_dead start 2024-09-13T16:48:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.884+0000 7f9be8c00640 1 -- 172.21.15.28:0/886170882 reap_dead start 2024-09-13T16:48:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.884+0000 7f9beac00640 1 -- 172.21.15.28:0/886170882 shutdown_connections 2024-09-13T16:48:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.884+0000 7f9beac00640 1 -- 172.21.15.28:0/886170882 >> 172.21.15.28:0/886170882 conn(0x7f9be40ca6b0 msgr2=0x7f9be40caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.884+0000 7f9beac00640 1 -- 172.21.15.28:0/886170882 shutdown_connections 2024-09-13T16:48:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:08.884+0000 7f9beac00640 1 -- 172.21.15.28:0/886170882 wait complete. 2024-09-13T16:48:08.904 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-13T16:48:08.904 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-13T16:48:08.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-13T16:48:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.058+0000 7f2da1c00640 1 Processor -- start 2024-09-13T16:48:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.058+0000 7f2da1c00640 1 -- start start 2024-09-13T16:48:09.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.059+0000 7f2da1c00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2d9c152f20 0x7f2d9c15c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:09.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.059+0000 7f2da1c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9c15c340 con 0x7f2d9c0c6a00 2024-09-13T16:48:09.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.059+0000 7f2da1c00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9c158fd0 con 0x7f2d9c158b20 2024-09-13T16:48:09.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.059+0000 7f2da1c00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d9c159b50 con 0x7f2d9c152f20 2024-09-13T16:48:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.060+0000 7f2d9aa00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2d9c152f20 0x7f2d9c15c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.060+0000 7f2d9b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2d9c0c6a00 0x7f2d9c1634a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37206/0 (socket says 172.21.15.28:37206) 2024-09-13T16:48:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.060+0000 7f2d9b400640 1 -- 172.21.15.28:0/3451018453 learned_addr learned my addr 172.21.15.28:0/3451018453 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.060+0000 7f2d9aa00640 1 -- 172.21.15.28:0/3451018453 >> v1:172.21.15.191:6789/0 conn(0x7f2d9c158b20 legacy=0x7f2d9c0c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.060+0000 7f2d9aa00640 1 -- 172.21.15.28:0/3451018453 >> v1:172.21.15.28:6789/0 conn(0x7f2d9c0c6a00 legacy=0x7f2d9c1634a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.060+0000 7f2d9aa00640 1 -- 172.21.15.28:0/3451018453 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9c1592b0 con 0x7f2d9c152f20 2024-09-13T16:48:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.061+0000 7f2d9aa00640 1 --2- 172.21.15.28:0/3451018453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d9c152f20 0x7f2d9c15c7d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0097a0 tx=0x7f2d8c02a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d524a3005c03cac4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.062+0000 7f2d9a000640 1 -- 172.21.15.28:0/3451018453 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c032640 con 0x7f2d9c152f20 2024-09-13T16:48:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.062+0000 7f2d9a000640 1 -- 172.21.15.28:0/3451018453 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8c0327a0 con 0x7f2d9c152f20 2024-09-13T16:48:09.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.062+0000 7f2d9a000640 1 -- 172.21.15.28:0/3451018453 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c032b30 con 0x7f2d9c152f20 2024-09-13T16:48:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.062+0000 7f2da1c00640 1 -- 172.21.15.28:0/3451018453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d9c152f20 msgr2=0x7f2d9c15c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.062+0000 7f2da1c00640 1 --2- 172.21.15.28:0/3451018453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d9c152f20 0x7f2d9c15c7d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0097a0 tx=0x7f2d8c02a2c0 comp rx=0 tx=0).stop 2024-09-13T16:48:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.062+0000 7f2da1c00640 1 -- 172.21.15.28:0/3451018453 shutdown_connections 2024-09-13T16:48:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.062+0000 7f2da1c00640 1 --2- 172.21.15.28:0/3451018453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d9c152f20 0x7f2d9c15c7d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.062+0000 7f2da1c00640 1 -- 172.21.15.28:0/3451018453 >> 172.21.15.28:0/3451018453 conn(0x7f2d9c066040 msgr2=0x7f2d9c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.062+0000 7f2da1c00640 1 -- 172.21.15.28:0/3451018453 shutdown_connections 2024-09-13T16:48:09.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.062+0000 7f2da1c00640 1 -- 172.21.15.28:0/3451018453 wait complete. 2024-09-13T16:48:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.064+0000 7f2da1c00640 1 Processor -- start 2024-09-13T16:48:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.064+0000 7f2da1c00640 1 -- start start 2024-09-13T16:48:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.065+0000 7f2da1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0c6a00 0x7f2d9c14ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.065+0000 7f2da1c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d9c152f20 0x7f2d9c070f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.065+0000 7f2da1c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d9c158b20 0x7f2d9c15a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.065+0000 7f2da1c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9c14f350 con 0x7f2d9c0c6a00 2024-09-13T16:48:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.065+0000 7f2da1c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9c15b920 con 0x7f2d9c152f20 2024-09-13T16:48:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.065+0000 7f2da1c00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9c15bc00 con 0x7f2d9c158b20 2024-09-13T16:48:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.065+0000 7f2d9aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0c6a00 0x7f2d9c14ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.065+0000 7f2d9aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0c6a00 0x7f2d9c14ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59572/0 (socket says 172.21.15.28:59572) 2024-09-13T16:48:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.065+0000 7f2d9aa00640 1 -- 172.21.15.28:0/732683439 learned_addr learned my addr 172.21.15.28:0/732683439 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.065+0000 7f2d93e00640 1 --2- 172.21.15.28:0/732683439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d9c152f20 0x7f2d9c070f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.065+0000 7f2d9b400640 1 --2- 172.21.15.28:0/732683439 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d9c158b20 0x7f2d9c15a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.065+0000 7f2d9aa00640 1 -- 172.21.15.28:0/732683439 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d9c158b20 msgr2=0x7f2d9c15a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.065+0000 7f2d9aa00640 1 --2- 172.21.15.28:0/732683439 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d9c158b20 0x7f2d9c15a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.065+0000 7f2d9aa00640 1 -- 172.21.15.28:0/732683439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d9c152f20 msgr2=0x7f2d9c070f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.065+0000 7f2d9aa00640 1 --2- 172.21.15.28:0/732683439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d9c152f20 0x7f2d9c070f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.065+0000 7f2d9aa00640 1 -- 172.21.15.28:0/732683439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c009400 con 0x7f2d9c0c6a00 2024-09-13T16:48:09.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.066+0000 7f2d9aa00640 1 --2- 172.21.15.28:0/732683439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0c6a00 0x7f2d9c14ebc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0098c0 tx=0x7f2d8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:09.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.066+0000 7f2d92a00640 1 -- 172.21.15.28:0/732683439 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c030980 con 0x7f2d9c0c6a00 2024-09-13T16:48:09.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.066+0000 7f2d92a00640 1 -- 172.21.15.28:0/732683439 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8c030b00 con 0x7f2d9c0c6a00 2024-09-13T16:48:09.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.066+0000 7f2d92a00640 1 -- 172.21.15.28:0/732683439 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c03d440 con 0x7f2d9c0c6a00 2024-09-13T16:48:09.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.066+0000 7f2da1c00640 1 -- 172.21.15.28:0/732683439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d9c14f8f0 con 0x7f2d9c0c6a00 2024-09-13T16:48:09.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.066+0000 7f2da1c00640 1 -- 172.21.15.28:0/732683439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d9c1f29f0 con 0x7f2d9c0c6a00 2024-09-13T16:48:09.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.067+0000 7f2da1c00640 1 -- 172.21.15.28:0/732683439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d9c0c5180 con 0x7f2d9c0c6a00 2024-09-13T16:48:09.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.068+0000 7f2d92a00640 1 -- 172.21.15.28:0/732683439 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2d8c039040 con 0x7f2d9c0c6a00 2024-09-13T16:48:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.069+0000 7f2d92a00640 1 --2- 172.21.15.28:0/732683439 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f2d7806ac00 0x7f2d7806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.069+0000 7f2d92a00640 1 -- 172.21.15.28:0/732683439 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2d8c032e40 con 0x7f2d9c0c6a00 2024-09-13T16:48:09.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.069+0000 7f2d93e00640 1 --2- 172.21.15.28:0/732683439 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f2d7806ac00 0x7f2d7806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.070+0000 7f2d93e00640 1 --2- 172.21.15.28:0/732683439 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f2d7806ac00 0x7f2d7806d0c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c1572c0 tx=0x7f2d84009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.074+0000 7f2d92a00640 1 -- 172.21.15.28:0/732683439 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2d8c075270 con 0x7f2d9c0c6a00 2024-09-13T16:48:09.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.324+0000 7f2da1c00640 1 -- 172.21.15.28:0/732683439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f2d9c071ce0 con 0x7f2d9c0c6a00 2024-09-13T16:48:09.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.325+0000 7f2d92a00640 1 -- 172.21.15.28:0/732683439 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f2d8c078890 con 0x7f2d9c0c6a00 2024-09-13T16:48:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.327+0000 7f2da1c00640 1 -- 172.21.15.28:0/732683439 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f2d7806ac00 msgr2=0x7f2d7806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.327+0000 7f2da1c00640 1 --2- 172.21.15.28:0/732683439 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f2d7806ac00 0x7f2d7806d0c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c1572c0 tx=0x7f2d84009040 comp rx=0 tx=0).stop 2024-09-13T16:48:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.327+0000 7f2da1c00640 1 -- 172.21.15.28:0/732683439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0c6a00 msgr2=0x7f2d9c14ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.327+0000 7f2da1c00640 1 --2- 172.21.15.28:0/732683439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0c6a00 0x7f2d9c14ebc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0098c0 tx=0x7f2d8c0046c0 comp rx=0 tx=0).stop 2024-09-13T16:48:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.328+0000 7f2da1c00640 1 -- 172.21.15.28:0/732683439 shutdown_connections 2024-09-13T16:48:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.328+0000 7f2da1c00640 1 --2- 172.21.15.28:0/732683439 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f2d7806ac00 0x7f2d7806d0c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.328+0000 7f2da1c00640 1 --2- 172.21.15.28:0/732683439 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d9c158b20 0x7f2d9c15a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.328+0000 7f2da1c00640 1 --2- 172.21.15.28:0/732683439 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d9c152f20 0x7f2d9c070f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.328+0000 7f2da1c00640 1 --2- 172.21.15.28:0/732683439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0c6a00 0x7f2d9c14ebc0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.328+0000 7f2da1c00640 1 -- 172.21.15.28:0/732683439 >> 172.21.15.28:0/732683439 conn(0x7f2d9c066040 msgr2=0x7f2d9c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.328+0000 7f2da1c00640 1 -- 172.21.15.28:0/732683439 shutdown_connections 2024-09-13T16:48:09.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:09.328+0000 7f2da1c00640 1 -- 172.21.15.28:0/732683439 wait complete. 2024-09-13T16:48:09.351 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-09-13T16:48:19.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:48:19.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.519+0000 7f913fe00640 1 Processor -- start 2024-09-13T16:48:19.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.520+0000 7f913fe00640 1 -- start start 2024-09-13T16:48:19.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.521+0000 7f913fe00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f91400c6240 0x7f9140162010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.521+0000 7f913fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914006e100 con 0x7f91400c1b40 2024-09-13T16:48:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.521+0000 7f913fe00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9140150d90 con 0x7f91401509b0 2024-09-13T16:48:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.521+0000 7f913fe00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91400c43b0 con 0x7f91400c6240 2024-09-13T16:48:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.521+0000 7f9137e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f91400c1b40 0x7f9140156990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52664/0 (socket says 172.21.15.28:52664) 2024-09-13T16:48:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.521+0000 7f9137e00640 1 -- 172.21.15.28:0/1589809641 learned_addr learned my addr 172.21.15.28:0/1589809641 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.521+0000 7f913f400640 1 --2- 172.21.15.28:0/1589809641 >> v2:172.21.15.191:3300/0 conn(0x7f91400c6240 0x7f9140162010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:19.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.522+0000 7f913e000640 1 -- 172.21.15.28:0/1589809641 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432266179 0 0) 0x7f914006e100 con 0x7f91400c1b40 2024-09-13T16:48:19.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.522+0000 7f913e000640 1 -- 172.21.15.28:0/1589809641 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9124003520 con 0x7f91400c1b40 2024-09-13T16:48:19.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.522+0000 7f913f400640 1 -- 172.21.15.28:0/1589809641 >> v1:172.21.15.191:6789/0 conn(0x7f91401509b0 legacy=0x7f914006cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:19.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.522+0000 7f913f400640 1 -- 172.21.15.28:0/1589809641 >> v1:172.21.15.28:6789/0 conn(0x7f91400c1b40 legacy=0x7f9140156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:19.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.522+0000 7f913f400640 1 -- 172.21.15.28:0/1589809641 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914006d7d0 con 0x7f91400c6240 2024-09-13T16:48:19.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.523+0000 7f913f400640 1 --2- 172.21.15.28:0/1589809641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91400c6240 0x7f9140162010 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9138009d90 tx=0x7f913802a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48ce395012f1d267 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:19.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.523+0000 7f913e000640 1 -- 172.21.15.28:0/1589809641 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9138007740 con 0x7f91400c6240 2024-09-13T16:48:19.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.523+0000 7f913e000640 1 -- 172.21.15.28:0/1589809641 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913802d050 con 0x7f91400c6240 2024-09-13T16:48:19.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.523+0000 7f913e000640 1 -- 172.21.15.28:0/1589809641 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9138007ac0 con 0x7f91400c6240 2024-09-13T16:48:19.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.523+0000 7f913fe00640 1 -- 172.21.15.28:0/1589809641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91400c6240 msgr2=0x7f9140162010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:19.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.523+0000 7f913fe00640 1 --2- 172.21.15.28:0/1589809641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91400c6240 0x7f9140162010 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9138009d90 tx=0x7f913802a6f0 comp rx=0 tx=0).stop 2024-09-13T16:48:19.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.524+0000 7f913fe00640 1 -- 172.21.15.28:0/1589809641 shutdown_connections 2024-09-13T16:48:19.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.524+0000 7f913fe00640 1 --2- 172.21.15.28:0/1589809641 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91400c6240 0x7f9140162010 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:19.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.524+0000 7f913fe00640 1 -- 172.21.15.28:0/1589809641 >> 172.21.15.28:0/1589809641 conn(0x7f91400ca0e0 msgr2=0x7f91400ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:19.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.524+0000 7f913fe00640 1 -- 172.21.15.28:0/1589809641 shutdown_connections 2024-09-13T16:48:19.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.524+0000 7f913fe00640 1 -- 172.21.15.28:0/1589809641 wait complete. 2024-09-13T16:48:19.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.525+0000 7f913fe00640 1 Processor -- start 2024-09-13T16:48:19.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.525+0000 7f913fe00640 1 -- start start 2024-09-13T16:48:19.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.526+0000 7f913fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91400c1b40 0x7f91400c5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:19.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.526+0000 7f913fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91400c6240 0x7f9140154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:19.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.526+0000 7f913fe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f91401509b0 0x7f914014ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:19.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.526+0000 7f913fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9140152cc0 con 0x7f91400c1b40 2024-09-13T16:48:19.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.526+0000 7f913fe00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91401535a0 con 0x7f91400c6240 2024-09-13T16:48:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.526+0000 7f913fe00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91401f2110 con 0x7f91401509b0 2024-09-13T16:48:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.526+0000 7f913ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91400c1b40 0x7f91400c5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.526+0000 7f9137e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91400c6240 0x7f9140154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.526+0000 7f9137e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91400c6240 0x7f9140154350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:38564/0 (socket says 172.21.15.28:38564) 2024-09-13T16:48:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.526+0000 7f9137e00640 1 -- 172.21.15.28:0/1085261934 learned_addr learned my addr 172.21.15.28:0/1085261934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.526+0000 7f913ea00640 1 --2- 172.21.15.28:0/1085261934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91400c1b40 0x7f91400c5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39064/0 (socket says 172.21.15.28:39064) 2024-09-13T16:48:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.527+0000 7f913f400640 1 --2- 172.21.15.28:0/1085261934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f91401509b0 0x7f914014ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.527+0000 7f9137e00640 1 -- 172.21.15.28:0/1085261934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f91401509b0 msgr2=0x7f914014ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.527+0000 7f9137e00640 1 --2- 172.21.15.28:0/1085261934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f91401509b0 0x7f914014ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.527+0000 7f9137e00640 1 -- 172.21.15.28:0/1085261934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91400c1b40 msgr2=0x7f91400c5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.527+0000 7f9137e00640 1 --2- 172.21.15.28:0/1085261934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91400c1b40 0x7f91400c5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.527+0000 7f9137e00640 1 -- 172.21.15.28:0/1085261934 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9138009a70 con 0x7f91400c6240 2024-09-13T16:48:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.527+0000 7f9137e00640 1 --2- 172.21.15.28:0/1085261934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91400c6240 0x7f9140154350 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f912800c440 tx=0x7f912800c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.527+0000 7f9136a00640 1 -- 172.21.15.28:0/1085261934 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9128005ce0 con 0x7f91400c6240 2024-09-13T16:48:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.527+0000 7f9136a00640 1 -- 172.21.15.28:0/1085261934 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9128005e40 con 0x7f91400c6240 2024-09-13T16:48:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.527+0000 7f913fe00640 1 -- 172.21.15.28:0/1085261934 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9140153a00 con 0x7f91400c6240 2024-09-13T16:48:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.527+0000 7f9136a00640 1 -- 172.21.15.28:0/1085261934 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91280072b0 con 0x7f91400c6240 2024-09-13T16:48:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.527+0000 7f913fe00640 1 -- 172.21.15.28:0/1085261934 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91401f17e0 con 0x7f91400c6240 2024-09-13T16:48:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.529+0000 7f913fe00640 1 -- 172.21.15.28:0/1085261934 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91400c2a00 con 0x7f91400c6240 2024-09-13T16:48:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.529+0000 7f9136a00640 1 -- 172.21.15.28:0/1085261934 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9128007410 con 0x7f91400c6240 2024-09-13T16:48:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.530+0000 7f9136a00640 1 --2- 172.21.15.28:0/1085261934 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f911c06ac00 0x7f911c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.530+0000 7f9136a00640 1 -- 172.21.15.28:0/1085261934 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f91280889d0 con 0x7f91400c6240 2024-09-13T16:48:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.530+0000 7f913ea00640 1 --2- 172.21.15.28:0/1085261934 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f911c06ac00 0x7f911c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:19.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.531+0000 7f913ea00640 1 --2- 172.21.15.28:0/1085261934 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f911c06ac00 0x7f911c06d0c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9130000f60 tx=0x7f9130009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:19.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.535+0000 7f9136a00640 1 -- 172.21.15.28:0/1085261934 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9128053c00 con 0x7f91400c6240 2024-09-13T16:48:19.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.751+0000 7f913fe00640 1 -- 172.21.15.28:0/1085261934 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91401f2250 con 0x7f91400c6240 2024-09-13T16:48:19.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.752+0000 7f9136a00640 1 -- 172.21.15.28:0/1085261934 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6307 (secure 0 0 0) 0x7f9128017020 con 0x7f91400c6240 2024-09-13T16:48:19.754 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:48:19.754 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4227,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/355570369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":355570369},{"type":"v1","addr":"172.21.15.28:6835","nonce":355570369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:43:54.969699+0000","modified":"2024-09-13T16:44:26.685528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4250,"mds_2":4244,"mds_3":4235,"mds_4":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"f","rank":4,"incarnation":16,"state":"up:active","state_seq":10,"addr":"172.21.15.191:6839/2366383263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6837","nonce":2366383263},{"type":"v1","addr":"172.21.15.191:6839","nonce":2366383263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4235":{"gid":4235,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.191:6838/2702860767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2702860767},{"type":"v1","addr":"172.21.15.191:6838","nonce":2702860767}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/2240589617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2240589617},{"type":"v1","addr":"172.21.15.28:6833","nonce":2240589617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6837/4068563017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":4068563017},{"type":"v1","addr":"172.21.15.28:6837","nonce":4068563017}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4258":{"gid":4258,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/704943585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":704943585},{"type":"v1","addr":"172.21.15.191:6835","nonce":704943585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T16:48:19.754 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-09-13T16:48:19.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.754+0000 7f913fe00640 1 -- 172.21.15.28:0/1085261934 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f911c06ac00 msgr2=0x7f911c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:19.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.754+0000 7f913fe00640 1 --2- 172.21.15.28:0/1085261934 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f911c06ac00 0x7f911c06d0c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9130000f60 tx=0x7f9130009040 comp rx=0 tx=0).stop 2024-09-13T16:48:19.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.754+0000 7f913fe00640 1 -- 172.21.15.28:0/1085261934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91400c6240 msgr2=0x7f9140154350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:19.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.755+0000 7f913fe00640 1 --2- 172.21.15.28:0/1085261934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91400c6240 0x7f9140154350 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f912800c440 tx=0x7f912800c900 comp rx=0 tx=0).stop 2024-09-13T16:48:19.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.755+0000 7f913fe00640 1 -- 172.21.15.28:0/1085261934 shutdown_connections 2024-09-13T16:48:19.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.755+0000 7f913fe00640 1 --2- 172.21.15.28:0/1085261934 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f911c06ac00 0x7f911c06d0c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:19.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.755+0000 7f913fe00640 1 --2- 172.21.15.28:0/1085261934 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f91401509b0 0x7f914014ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:19.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.755+0000 7f913fe00640 1 --2- 172.21.15.28:0/1085261934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f91400c6240 0x7f9140154350 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:19.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.755+0000 7f913fe00640 1 --2- 172.21.15.28:0/1085261934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91400c1b40 0x7f91400c5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:19.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.755+0000 7f913fe00640 1 -- 172.21.15.28:0/1085261934 >> 172.21.15.28:0/1085261934 conn(0x7f91400ca0e0 msgr2=0x7f914014d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:19.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.755+0000 7f913fe00640 1 -- 172.21.15.28:0/1085261934 shutdown_connections 2024-09-13T16:48:19.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.755+0000 7f913fe00640 1 -- 172.21.15.28:0/1085261934 wait complete. 2024-09-13T16:48:19.773 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-13T16:48:19.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T16:48:19.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.926+0000 7f528c800640 1 Processor -- start 2024-09-13T16:48:19.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.926+0000 7f528c800640 1 -- start start 2024-09-13T16:48:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.928+0000 7f528c800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f528414fa80 0x7f52840c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.928+0000 7f528c800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52840c6350 0x7f52841559d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.928+0000 7f528c800640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52840c42b0 con 0x7f5284072170 2024-09-13T16:48:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.928+0000 7f528c800640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52840c36e0 con 0x7f52840c6350 2024-09-13T16:48:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.928+0000 7f528c800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52840c3980 con 0x7f528414fa80 2024-09-13T16:48:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.928+0000 7f5289400640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f5284072170 0x7f52840c6ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:41356/0 (socket says 172.21.15.28:41356) 2024-09-13T16:48:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.928+0000 7f5289400640 1 -- 172.21.15.28:0/1893787255 learned_addr learned my addr 172.21.15.28:0/1893787255 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.928+0000 7f5289e00640 1 --2- 172.21.15.28:0/1893787255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f528414fa80 0x7f52840c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.928+0000 7f528a800640 1 --2- 172.21.15.28:0/1893787255 >> v2:172.21.15.191:3300/0 conn(0x7f52840c6350 0x7f52841559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.928+0000 7f5288a00640 1 -- 172.21.15.28:0/1893787255 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564517741 0 0) 0x7f52840c42b0 con 0x7f5284072170 2024-09-13T16:48:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.928+0000 7f5288a00640 1 -- 172.21.15.28:0/1893787255 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f526c003520 con 0x7f5284072170 2024-09-13T16:48:19.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.928+0000 7f5289e00640 1 -- 172.21.15.28:0/1893787255 >> v2:172.21.15.191:3300/0 conn(0x7f52840c6350 msgr2=0x7f52841559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:19.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.928+0000 7f5289e00640 1 --2- 172.21.15.28:0/1893787255 >> v2:172.21.15.191:3300/0 conn(0x7f52840c6350 0x7f52841559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:19.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.928+0000 7f5289e00640 1 -- 172.21.15.28:0/1893787255 >> v1:172.21.15.191:6789/0 conn(0x7f5284072170 legacy=0x7f52840c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:19.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.928+0000 7f5289e00640 1 -- 172.21.15.28:0/1893787255 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52840c3c60 con 0x7f528414fa80 2024-09-13T16:48:19.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.930+0000 7f5289e00640 1 --2- 172.21.15.28:0/1893787255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f528414fa80 0x7f52840c1960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f52780097a0 tx=0x7f527802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=958379b57f4af0a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:19.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.930+0000 7f5288a00640 1 -- 172.21.15.28:0/1893787255 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278032440 con 0x7f528414fa80 2024-09-13T16:48:19.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.930+0000 7f5288a00640 1 -- 172.21.15.28:0/1893787255 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52780325a0 con 0x7f528414fa80 2024-09-13T16:48:19.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.930+0000 7f5288a00640 1 -- 172.21.15.28:0/1893787255 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278032930 con 0x7f528414fa80 2024-09-13T16:48:19.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.930+0000 7f528c800640 1 -- 172.21.15.28:0/1893787255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f528414fa80 msgr2=0x7f52840c1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:19.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.930+0000 7f528c800640 1 --2- 172.21.15.28:0/1893787255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f528414fa80 0x7f52840c1960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f52780097a0 tx=0x7f527802a180 comp rx=0 tx=0).stop 2024-09-13T16:48:19.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.931+0000 7f528c800640 1 -- 172.21.15.28:0/1893787255 shutdown_connections 2024-09-13T16:48:19.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.931+0000 7f528c800640 1 --2- 172.21.15.28:0/1893787255 >> v2:172.21.15.191:3300/0 conn(0x7f52840c6350 0x7f52841559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:19.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.931+0000 7f528c800640 1 --2- 172.21.15.28:0/1893787255 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f528414fa80 0x7f52840c1960 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:19.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.931+0000 7f528c800640 1 -- 172.21.15.28:0/1893787255 >> 172.21.15.28:0/1893787255 conn(0x7f5284069e80 msgr2=0x7f528406a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:19.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.931+0000 7f528c800640 1 -- 172.21.15.28:0/1893787255 shutdown_connections 2024-09-13T16:48:19.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.931+0000 7f528c800640 1 -- 172.21.15.28:0/1893787255 wait complete. 2024-09-13T16:48:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.932+0000 7f528c800640 1 Processor -- start 2024-09-13T16:48:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.932+0000 7f528c800640 1 -- start start 2024-09-13T16:48:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.933+0000 7f528c800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5284072170 0x7f52841501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.933+0000 7f528c800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52840c6350 0x7f5284141ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.933+0000 7f528c800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f528414fa80 0x7f52841f0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.933+0000 7f528c800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52841efdf0 con 0x7f52840c6350 2024-09-13T16:48:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.933+0000 7f528c800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52841ef7c0 con 0x7f5284072170 2024-09-13T16:48:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.933+0000 7f528c800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5284152220 con 0x7f528414fa80 2024-09-13T16:48:19.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.934+0000 7f5289400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52840c6350 0x7f5284141ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:19.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.934+0000 7f5289400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52840c6350 0x7f5284141ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39066/0 (socket says 172.21.15.28:39066) 2024-09-13T16:48:19.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.934+0000 7f5289e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5284072170 0x7f52841501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:19.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.934+0000 7f5289400640 1 -- 172.21.15.28:0/1013423110 learned_addr learned my addr 172.21.15.28:0/1013423110 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:19.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.934+0000 7f528a800640 1 --2- 172.21.15.28:0/1013423110 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f528414fa80 0x7f52841f0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:19.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.934+0000 7f5289e00640 1 -- 172.21.15.28:0/1013423110 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f528414fa80 msgr2=0x7f52841f0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:19.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.934+0000 7f5289e00640 1 --2- 172.21.15.28:0/1013423110 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f528414fa80 0x7f52841f0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:19.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.934+0000 7f5289e00640 1 -- 172.21.15.28:0/1013423110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52840c6350 msgr2=0x7f5284141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:19.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.934+0000 7f5289e00640 1 --2- 172.21.15.28:0/1013423110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52840c6350 0x7f5284141ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:19.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.934+0000 7f5289e00640 1 -- 172.21.15.28:0/1013423110 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5278009400 con 0x7f5284072170 2024-09-13T16:48:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.934+0000 7f5289e00640 1 --2- 172.21.15.28:0/1013423110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5284072170 0x7f52841501d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5278031000 tx=0x7f52780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.935+0000 7f527e000640 1 -- 172.21.15.28:0/1013423110 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278030b80 con 0x7f5284072170 2024-09-13T16:48:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.935+0000 7f527e000640 1 -- 172.21.15.28:0/1013423110 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5278030ce0 con 0x7f5284072170 2024-09-13T16:48:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.935+0000 7f528c800640 1 -- 172.21.15.28:0/1013423110 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5284150860 con 0x7f5284072170 2024-09-13T16:48:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.935+0000 7f527e000640 1 -- 172.21.15.28:0/1013423110 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527803c640 con 0x7f5284072170 2024-09-13T16:48:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.935+0000 7f528c800640 1 -- 172.21.15.28:0/1013423110 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5284150dc0 con 0x7f5284072170 2024-09-13T16:48:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.936+0000 7f528c800640 1 -- 172.21.15.28:0/1013423110 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52841f0060 con 0x7f5284072170 2024-09-13T16:48:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.937+0000 7f527e000640 1 -- 172.21.15.28:0/1013423110 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f527803c7a0 con 0x7f5284072170 2024-09-13T16:48:19.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.938+0000 7f527e000640 1 --2- 172.21.15.28:0/1013423110 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f526806ac30 0x7f526806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:19.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.938+0000 7f527e000640 1 -- 172.21.15.28:0/1013423110 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f52780aa500 con 0x7f5284072170 2024-09-13T16:48:19.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.938+0000 7f5289400640 1 --2- 172.21.15.28:0/1013423110 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f526806ac30 0x7f526806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:19.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.939+0000 7f5289400640 1 --2- 172.21.15.28:0/1013423110 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f526806ac30 0x7f526806d0f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5284142930 tx=0x7f5274009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:19.941+0000 7f527e000640 1 -- 172.21.15.28:0/1013423110 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5278075780 con 0x7f5284072170 2024-09-13T16:48:20.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.189+0000 7f528c800640 1 -- 172.21.15.28:0/1013423110 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f52840c1960 con 0x7f5284072170 2024-09-13T16:48:20.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.189+0000 7f527e000640 1 -- 172.21.15.28:0/1013423110 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1548 (secure 0 0 0) 0x7f5278037080 con 0x7f5284072170 2024-09-13T16:48:20.191 INFO:teuthology.orchestra.run.smithi028.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","modified":"2024-09-13T16:43:05.305835Z","created":"2024-09-13T16:43:05.305835Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3301","nonce":0},{"type":"v1","addr":"172.21.15.191:6790","nonce":0}]},"addr":"172.21.15.191:6790/0","public_addr":"172.21.15.191:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T16:48:20.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.192+0000 7f528c800640 1 -- 172.21.15.28:0/1013423110 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f526806ac30 msgr2=0x7f526806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:20.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.192+0000 7f528c800640 1 --2- 172.21.15.28:0/1013423110 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f526806ac30 0x7f526806d0f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5284142930 tx=0x7f5274009040 comp rx=0 tx=0).stop 2024-09-13T16:48:20.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.192+0000 7f528c800640 1 -- 172.21.15.28:0/1013423110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5284072170 msgr2=0x7f52841501d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:20.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.192+0000 7f528c800640 1 --2- 172.21.15.28:0/1013423110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5284072170 0x7f52841501d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5278031000 tx=0x7f52780046c0 comp rx=0 tx=0).stop 2024-09-13T16:48:20.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.192+0000 7f528c800640 1 -- 172.21.15.28:0/1013423110 shutdown_connections 2024-09-13T16:48:20.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.192+0000 7f528c800640 1 --2- 172.21.15.28:0/1013423110 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f526806ac30 0x7f526806d0f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:20.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.192+0000 7f528c800640 1 --2- 172.21.15.28:0/1013423110 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f528414fa80 0x7f52841f0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:20.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.192+0000 7f528c800640 1 --2- 172.21.15.28:0/1013423110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52840c6350 0x7f5284141ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:20.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.192+0000 7f528c800640 1 --2- 172.21.15.28:0/1013423110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5284072170 0x7f52841501d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:20.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.192+0000 7f528c800640 1 -- 172.21.15.28:0/1013423110 >> 172.21.15.28:0/1013423110 conn(0x7f5284069e80 msgr2=0x7f52840c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:20.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.192+0000 7f528c800640 1 -- 172.21.15.28:0/1013423110 shutdown_connections 2024-09-13T16:48:20.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.192+0000 7f528c800640 1 -- 172.21.15.28:0/1013423110 wait complete. 2024-09-13T16:48:20.210 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-13T16:48:20.210 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-13T16:48:20.220 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-13T16:48:20.220 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:48:20.220 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:48:20.256 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T16:48:20.257 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:48:20.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:48:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.527+0000 7fe9c9a00640 1 Processor -- start 2024-09-13T16:48:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.527+0000 7fe9c9a00640 1 -- start start 2024-09-13T16:48:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.528+0000 7fe9c9a00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe9c40c2250 0x7fe9c4162230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.528+0000 7fe9c9a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c4151080 con 0x7fe9c4150ca0 2024-09-13T16:48:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.528+0000 7fe9c9a00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c406a990 con 0x7fe9c41568e0 2024-09-13T16:48:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.528+0000 7fe9c9a00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c40c2680 con 0x7fe9c40c2250 2024-09-13T16:48:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.528+0000 7fe9c2a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe9c4150ca0 0x7fe9c415a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52678/0 (socket says 172.21.15.28:52678) 2024-09-13T16:48:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.528+0000 7fe9c2a00640 1 -- 172.21.15.28:0/1890014519 learned_addr learned my addr 172.21.15.28:0/1890014519 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.528+0000 7fe9c3400640 1 --2- 172.21.15.28:0/1890014519 >> v2:172.21.15.191:3300/0 conn(0x7fe9c40c2250 0x7fe9c4162230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.529+0000 7fe9c2000640 1 -- 172.21.15.28:0/1890014519 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182396773 0 0) 0x7fe9c4151080 con 0x7fe9c4150ca0 2024-09-13T16:48:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.529+0000 7fe9c2000640 1 -- 172.21.15.28:0/1890014519 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9a8003520 con 0x7fe9c4150ca0 2024-09-13T16:48:20.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.529+0000 7fe9c3400640 1 -- 172.21.15.28:0/1890014519 >> v1:172.21.15.191:6789/0 conn(0x7fe9c41568e0 legacy=0x7fe9c40c4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:20.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.529+0000 7fe9c3400640 1 -- 172.21.15.28:0/1890014519 >> v1:172.21.15.28:6789/0 conn(0x7fe9c4150ca0 legacy=0x7fe9c415a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:20.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.529+0000 7fe9c3400640 1 -- 172.21.15.28:0/1890014519 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c40c2870 con 0x7fe9c40c2250 2024-09-13T16:48:20.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.530+0000 7fe9c3400640 1 --2- 172.21.15.28:0/1890014519 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c40c2250 0x7fe9c4162230 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009d90 tx=0x7fe9bc02a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1a542f0eb097013 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:20.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.530+0000 7fe9c2000640 1 -- 172.21.15.28:0/1890014519 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc007740 con 0x7fe9c40c2250 2024-09-13T16:48:20.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.530+0000 7fe9c2000640 1 -- 172.21.15.28:0/1890014519 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc02d050 con 0x7fe9c40c2250 2024-09-13T16:48:20.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.530+0000 7fe9c2000640 1 -- 172.21.15.28:0/1890014519 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc007ac0 con 0x7fe9c40c2250 2024-09-13T16:48:20.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.531+0000 7fe9c9a00640 1 -- 172.21.15.28:0/1890014519 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c40c2250 msgr2=0x7fe9c4162230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:20.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.531+0000 7fe9c9a00640 1 --2- 172.21.15.28:0/1890014519 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c40c2250 0x7fe9c4162230 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009d90 tx=0x7fe9bc02a660 comp rx=0 tx=0).stop 2024-09-13T16:48:20.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.531+0000 7fe9c9a00640 1 -- 172.21.15.28:0/1890014519 shutdown_connections 2024-09-13T16:48:20.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.531+0000 7fe9c9a00640 1 --2- 172.21.15.28:0/1890014519 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c40c2250 0x7fe9c4162230 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:20.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.531+0000 7fe9c9a00640 1 -- 172.21.15.28:0/1890014519 >> 172.21.15.28:0/1890014519 conn(0x7fe9c4066040 msgr2=0x7fe9c4069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:20.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.531+0000 7fe9c9a00640 1 -- 172.21.15.28:0/1890014519 shutdown_connections 2024-09-13T16:48:20.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.531+0000 7fe9c9a00640 1 -- 172.21.15.28:0/1890014519 wait complete. 2024-09-13T16:48:20.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.532+0000 7fe9c9a00640 1 Processor -- start 2024-09-13T16:48:20.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.533+0000 7fe9c9a00640 1 -- start start 2024-09-13T16:48:20.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.533+0000 7fe9c9a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c40c2250 0x7fe9c40718d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:20.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.533+0000 7fe9c9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4150ca0 0x7fe9c414c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:20.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.533+0000 7fe9c9a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9c41568e0 0x7fe9c41f1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:20.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.534+0000 7fe9c9a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c41e1570 con 0x7fe9c4150ca0 2024-09-13T16:48:20.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.534+0000 7fe9c9a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c41e1290 con 0x7fe9c40c2250 2024-09-13T16:48:20.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.534+0000 7fe9c9a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c41e2040 con 0x7fe9c41568e0 2024-09-13T16:48:20.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.534+0000 7fe9c2a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c40c2250 0x7fe9c40718d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:20.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.534+0000 7fe9bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4150ca0 0x7fe9c414c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:20.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.534+0000 7fe9c2a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c40c2250 0x7fe9c40718d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:38604/0 (socket says 172.21.15.28:38604) 2024-09-13T16:48:20.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.534+0000 7fe9c2a00640 1 -- 172.21.15.28:0/1068971177 learned_addr learned my addr 172.21.15.28:0/1068971177 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:20.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.534+0000 7fe9c3400640 1 --2- 172.21.15.28:0/1068971177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9c41568e0 0x7fe9c41f1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:20.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.534+0000 7fe9c2a00640 1 -- 172.21.15.28:0/1068971177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9c41568e0 msgr2=0x7fe9c41f1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:20.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.534+0000 7fe9c2a00640 1 --2- 172.21.15.28:0/1068971177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9c41568e0 0x7fe9c41f1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:20.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.534+0000 7fe9c2a00640 1 -- 172.21.15.28:0/1068971177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4150ca0 msgr2=0x7fe9c414c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:20.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.534+0000 7fe9c2a00640 1 --2- 172.21.15.28:0/1068971177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4150ca0 0x7fe9c414c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:20.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.534+0000 7fe9c2a00640 1 -- 172.21.15.28:0/1068971177 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc009a70 con 0x7fe9c40c2250 2024-09-13T16:48:20.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.534+0000 7fe9bbe00640 1 --2- 172.21.15.28:0/1068971177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4150ca0 0x7fe9c414c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:20.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.534+0000 7fe9c3400640 1 --2- 172.21.15.28:0/1068971177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9c41568e0 0x7fe9c41f1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:20.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.535+0000 7fe9c2a00640 1 --2- 172.21.15.28:0/1068971177 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c40c2250 0x7fe9c40718d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe9b400c500 tx=0x7fe9b400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:20.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.535+0000 7fe9baa00640 1 -- 172.21.15.28:0/1068971177 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4005ce0 con 0x7fe9c40c2250 2024-09-13T16:48:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.535+0000 7fe9baa00640 1 -- 172.21.15.28:0/1068971177 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b4005e40 con 0x7fe9c40c2250 2024-09-13T16:48:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.535+0000 7fe9baa00640 1 -- 172.21.15.28:0/1068971177 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4007330 con 0x7fe9c40c2250 2024-09-13T16:48:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.535+0000 7fe9c9a00640 1 -- 172.21.15.28:0/1068971177 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c41e17c0 con 0x7fe9c40c2250 2024-09-13T16:48:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.535+0000 7fe9c9a00640 1 -- 172.21.15.28:0/1068971177 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c41e1d60 con 0x7fe9c40c2250 2024-09-13T16:48:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.536+0000 7fe9c9a00640 1 -- 172.21.15.28:0/1068971177 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c415a210 con 0x7fe9c40c2250 2024-09-13T16:48:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.537+0000 7fe9baa00640 1 -- 172.21.15.28:0/1068971177 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe9b4007550 con 0x7fe9c40c2250 2024-09-13T16:48:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.538+0000 7fe9baa00640 1 --2- 172.21.15.28:0/1068971177 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe9a006ac00 0x7fe9a006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.538+0000 7fe9baa00640 1 -- 172.21.15.28:0/1068971177 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe9b4088a10 con 0x7fe9c40c2250 2024-09-13T16:48:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.538+0000 7fe9bbe00640 1 --2- 172.21.15.28:0/1068971177 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe9a006ac00 0x7fe9a006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.539+0000 7fe9bbe00640 1 --2- 172.21.15.28:0/1068971177 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe9a006ac00 0x7fe9a006d0c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe9c40c9ee0 tx=0x7fe9ac009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.543+0000 7fe9baa00640 1 -- 172.21.15.28:0/1068971177 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe9b4053cd0 con 0x7fe9c40c2250 2024-09-13T16:48:20.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.720+0000 7fe9c9a00640 1 -- 172.21.15.28:0/1068971177 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9c40c5910 con 0x7fe9c40c2250 2024-09-13T16:48:20.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.722+0000 7fe9baa00640 1 -- 172.21.15.28:0/1068971177 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16673 (secure 0 0 0) 0x7fe9b4017020 con 0x7fe9c40c2250 2024-09-13T16:48:20.723 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:48:20.724 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":22,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:44:58.950267+0000","last_up_change":"2024-09-13T16:43:49.476313+0000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:43:49.643606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:43:52.828510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:43:53.945874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6801","nonce":2964323435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6803","nonce":2964323435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6807","nonce":2964323435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6805","nonce":2964323435}]},"public_addr":"172.21.15.28:6801/2964323435","cluster_addr":"172.21.15.28:6803/2964323435","heartbeat_back_addr":"172.21.15.28:6807/2964323435","heartbeat_front_addr":"172.21.15.28:6805/2964323435","state":["exists","up"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6809","nonce":23768570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6811","nonce":23768570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6815","nonce":23768570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6813","nonce":23768570}]},"public_addr":"172.21.15.28:6809/23768570","cluster_addr":"172.21.15.28:6811/23768570","heartbeat_back_addr":"172.21.15.28:6815/23768570","heartbeat_front_addr":"172.21.15.28:6813/23768570","state":["exists","up"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6817","nonce":886887303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6819","nonce":886887303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6823","nonce":886887303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6821","nonce":886887303}]},"public_addr":"172.21.15.28:6817/886887303","cluster_addr":"172.21.15.28:6819/886887303","heartbeat_back_addr":"172.21.15.28:6823/886887303","heartbeat_front_addr":"172.21.15.28:6821/886887303","state":["exists","up"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6825","nonce":2616036674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6827","nonce":2616036674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6831","nonce":2616036674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6829","nonce":2616036674}]},"public_addr":"172.21.15.28:6825/2616036674","cluster_addr":"172.21.15.28:6827/2616036674","heartbeat_back_addr":"172.21.15.28:6831/2616036674","heartbeat_front_addr":"172.21.15.28:6829/2616036674","state":["exists","up"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6801","nonce":2817102450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6803","nonce":2817102450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6807","nonce":2817102450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6805","nonce":2817102450}]},"public_addr":"172.21.15.191:6801/2817102450","cluster_addr":"172.21.15.191:6803/2817102450","heartbeat_back_addr":"172.21.15.191:6807/2817102450","heartbeat_front_addr":"172.21.15.191:6805/2817102450","state":["exists","up"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6817","nonce":2503027971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6819","nonce":2503027971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6823","nonce":2503027971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6821","nonce":2503027971}]},"public_addr":"172.21.15.191:6817/2503027971","cluster_addr":"172.21.15.191:6819/2503027971","heartbeat_back_addr":"172.21.15.191:6823/2503027971","heartbeat_front_addr":"172.21.15.191:6821/2503027971","state":["exists","up"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6809","nonce":3006595088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6811","nonce":3006595088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6815","nonce":3006595088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6813","nonce":3006595088}]},"public_addr":"172.21.15.191:6809/3006595088","cluster_addr":"172.21.15.191:6811/3006595088","heartbeat_back_addr":"172.21.15.191:6815/3006595088","heartbeat_front_addr":"172.21.15.191:6813/3006595088","state":["exists","up"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6825","nonce":1477327720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6827","nonce":1477327720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6831","nonce":1477327720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6829","nonce":1477327720}]},"public_addr":"172.21.15.191:6825/1477327720","cluster_addr":"172.21.15.191:6827/1477327720","heartbeat_back_addr":"172.21.15.191:6831/1477327720","heartbeat_front_addr":"172.21.15.191:6829/1477327720","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.917250+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.954351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.017329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.046221+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.707467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.726947+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.782321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.764850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:48:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.724+0000 7fe9c9a00640 1 -- 172.21.15.28:0/1068971177 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe9a006ac00 msgr2=0x7fe9a006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.724+0000 7fe9c9a00640 1 --2- 172.21.15.28:0/1068971177 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe9a006ac00 0x7fe9a006d0c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe9c40c9ee0 tx=0x7fe9ac009040 comp rx=0 tx=0).stop 2024-09-13T16:48:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.724+0000 7fe9c9a00640 1 -- 172.21.15.28:0/1068971177 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c40c2250 msgr2=0x7fe9c40718d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.724+0000 7fe9c9a00640 1 --2- 172.21.15.28:0/1068971177 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c40c2250 0x7fe9c40718d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe9b400c500 tx=0x7fe9b400c9c0 comp rx=0 tx=0).stop 2024-09-13T16:48:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.724+0000 7fe9c9a00640 1 -- 172.21.15.28:0/1068971177 shutdown_connections 2024-09-13T16:48:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.724+0000 7fe9c9a00640 1 --2- 172.21.15.28:0/1068971177 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe9a006ac00 0x7fe9a006d0c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.724+0000 7fe9c9a00640 1 --2- 172.21.15.28:0/1068971177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9c41568e0 0x7fe9c41f1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.724+0000 7fe9c9a00640 1 --2- 172.21.15.28:0/1068971177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4150ca0 0x7fe9c414c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.724+0000 7fe9c9a00640 1 --2- 172.21.15.28:0/1068971177 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c40c2250 0x7fe9c40718d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.724+0000 7fe9c9a00640 1 -- 172.21.15.28:0/1068971177 >> 172.21.15.28:0/1068971177 conn(0x7fe9c4066040 msgr2=0x7fe9c40c4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.724+0000 7fe9c9a00640 1 -- 172.21.15.28:0/1068971177 shutdown_connections 2024-09-13T16:48:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.724+0000 7fe9c9a00640 1 -- 172.21.15.28:0/1068971177 wait complete. 2024-09-13T16:48:20.743 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:43:49.643606+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:43:52.828510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:43:53.945874+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:48:20.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:48:20.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.900+0000 7f1fd3a00640 1 Processor -- start 2024-09-13T16:48:20.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.900+0000 7f1fd3a00640 1 -- start start 2024-09-13T16:48:20.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.901+0000 7f1fd3a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1fcc0c83d0 0x7f1fcc0c9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:20.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.901+0000 7f1fd3a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc06e7f0 con 0x7f1fcc0cafb0 2024-09-13T16:48:20.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.901+0000 7f1fd3a00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc06e570 con 0x7f1fcc150b30 2024-09-13T16:48:20.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.901+0000 7f1fd3a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fcc0c5ac0 con 0x7f1fcc0c83d0 2024-09-13T16:48:20.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.902+0000 7f1fc3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1fcc0cafb0 0x7f1fcc15f0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52680/0 (socket says 172.21.15.28:52680) 2024-09-13T16:48:20.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.902+0000 7f1fc3e00640 1 -- 172.21.15.28:0/774448748 learned_addr learned my addr 172.21.15.28:0/774448748 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:20.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.902+0000 7f1fd1a00640 1 --2- 172.21.15.28:0/774448748 >> v2:172.21.15.28:3300/0 conn(0x7f1fcc0c83d0 0x7f1fcc0c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:20.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.902+0000 7f1fcbe00640 1 -- 172.21.15.28:0/774448748 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322751734 0 0) 0x7f1fcc06e570 con 0x7f1fcc150b30 2024-09-13T16:48:20.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.902+0000 7f1fcbe00640 1 -- 172.21.15.28:0/774448748 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb0003520 con 0x7f1fcc150b30 2024-09-13T16:48:20.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.902+0000 7f1fcbe00640 1 -- 172.21.15.28:0/774448748 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037237321 0 0) 0x7f1fcc06e7f0 con 0x7f1fcc0cafb0 2024-09-13T16:48:20.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.902+0000 7f1fcbe00640 1 -- 172.21.15.28:0/774448748 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fcc06e570 con 0x7f1fcc0cafb0 2024-09-13T16:48:20.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.902+0000 7f1fd1a00640 1 -- 172.21.15.28:0/774448748 >> v1:172.21.15.191:6789/0 conn(0x7f1fcc150b30 legacy=0x7f1fcc0c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:20.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.902+0000 7f1fd1a00640 1 -- 172.21.15.28:0/774448748 >> v1:172.21.15.28:6789/0 conn(0x7f1fcc0cafb0 legacy=0x7f1fcc15f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:20.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.902+0000 7f1fd1a00640 1 -- 172.21.15.28:0/774448748 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc1573c0 con 0x7f1fcc0c83d0 2024-09-13T16:48:20.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.903+0000 7f1fd1a00640 1 --2- 172.21.15.28:0/774448748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fcc0c83d0 0x7f1fcc0c9970 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009d90 tx=0x7f1fc402a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c0cb754fdb5f4a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:20.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.904+0000 7f1fcbe00640 1 -- 172.21.15.28:0/774448748 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4007740 con 0x7f1fcc0c83d0 2024-09-13T16:48:20.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.904+0000 7f1fcbe00640 1 -- 172.21.15.28:0/774448748 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc40078a0 con 0x7f1fcc0c83d0 2024-09-13T16:48:20.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.904+0000 7f1fcbe00640 1 -- 172.21.15.28:0/774448748 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4007c30 con 0x7f1fcc0c83d0 2024-09-13T16:48:20.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.904+0000 7f1fd3a00640 1 -- 172.21.15.28:0/774448748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fcc0c83d0 msgr2=0x7f1fcc0c9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:20.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.904+0000 7f1fd3a00640 1 --2- 172.21.15.28:0/774448748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fcc0c83d0 0x7f1fcc0c9970 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009d90 tx=0x7f1fc402a780 comp rx=0 tx=0).stop 2024-09-13T16:48:20.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.904+0000 7f1fd3a00640 1 -- 172.21.15.28:0/774448748 shutdown_connections 2024-09-13T16:48:20.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.904+0000 7f1fd3a00640 1 --2- 172.21.15.28:0/774448748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fcc0c83d0 0x7f1fcc0c9970 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:20.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.904+0000 7f1fd3a00640 1 -- 172.21.15.28:0/774448748 >> 172.21.15.28:0/774448748 conn(0x7f1fcc0ca2f0 msgr2=0x7f1fcc0ca6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:20.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.904+0000 7f1fd3a00640 1 -- 172.21.15.28:0/774448748 shutdown_connections 2024-09-13T16:48:20.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.904+0000 7f1fd3a00640 1 -- 172.21.15.28:0/774448748 wait complete. 2024-09-13T16:48:20.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.906+0000 7f1fd3a00640 1 Processor -- start 2024-09-13T16:48:20.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.906+0000 7f1fd3a00640 1 -- start start 2024-09-13T16:48:20.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.907+0000 7f1fd3a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1fcc0c83d0 0x7f1fcc146c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:20.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.907+0000 7f1fd3a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fcc0cafb0 0x7f1fcc14c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:20.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.907+0000 7f1fd3a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fcc150b30 0x7f1fcc1de3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:20.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.907+0000 7f1fd3a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc0c9eb0 con 0x7f1fcc0cafb0 2024-09-13T16:48:20.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.907+0000 7f1fd3a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc1dd990 con 0x7f1fcc150b30 2024-09-13T16:48:20.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.907+0000 7f1fd3a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc1ddf50 con 0x7f1fcc0c83d0 2024-09-13T16:48:20.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.907+0000 7f1fc3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fcc0cafb0 0x7f1fcc14c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:20.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.907+0000 7f1fc3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fcc0cafb0 0x7f1fcc14c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39100/0 (socket says 172.21.15.28:39100) 2024-09-13T16:48:20.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.907+0000 7f1fc3e00640 1 -- 172.21.15.28:0/2283417501 learned_addr learned my addr 172.21.15.28:0/2283417501 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:20.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.907+0000 7f1fd1a00640 1 --2- 172.21.15.28:0/2283417501 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fcc150b30 0x7f1fcc1de3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:20.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.907+0000 7f1fd1000640 1 --2- 172.21.15.28:0/2283417501 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1fcc0c83d0 0x7f1fcc146c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:20.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.908+0000 7f1fc3e00640 1 -- 172.21.15.28:0/2283417501 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1fcc0c83d0 msgr2=0x7f1fcc146c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:20.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.908+0000 7f1fc3e00640 1 --2- 172.21.15.28:0/2283417501 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1fcc0c83d0 0x7f1fcc146c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:20.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.908+0000 7f1fc3e00640 1 -- 172.21.15.28:0/2283417501 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fcc150b30 msgr2=0x7f1fcc1de3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:20.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.908+0000 7f1fc3e00640 1 --2- 172.21.15.28:0/2283417501 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fcc150b30 0x7f1fcc1de3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:20.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.908+0000 7f1fc3e00640 1 -- 172.21.15.28:0/2283417501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc4009a70 con 0x7f1fcc0cafb0 2024-09-13T16:48:20.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.908+0000 7f1fc3e00640 1 --2- 172.21.15.28:0/2283417501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fcc0cafb0 0x7f1fcc14c040 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1fb400c500 tx=0x7f1fb400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:20.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.908+0000 7f1fc9600640 1 -- 172.21.15.28:0/2283417501 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4005ce0 con 0x7f1fcc0cafb0 2024-09-13T16:48:20.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.908+0000 7f1fc9600640 1 -- 172.21.15.28:0/2283417501 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb4005e40 con 0x7f1fcc0cafb0 2024-09-13T16:48:20.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.908+0000 7f1fc9600640 1 -- 172.21.15.28:0/2283417501 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4007310 con 0x7f1fcc0cafb0 2024-09-13T16:48:20.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.908+0000 7f1fd3a00640 1 -- 172.21.15.28:0/2283417501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fcc071e10 con 0x7f1fcc0cafb0 2024-09-13T16:48:20.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.908+0000 7f1fd3a00640 1 -- 172.21.15.28:0/2283417501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fcc1f2240 con 0x7f1fcc0cafb0 2024-09-13T16:48:20.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.909+0000 7f1fd3a00640 1 -- 172.21.15.28:0/2283417501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fcc1576a0 con 0x7f1fcc0cafb0 2024-09-13T16:48:20.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.910+0000 7f1fc9600640 1 -- 172.21.15.28:0/2283417501 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1fb4007470 con 0x7f1fcc0cafb0 2024-09-13T16:48:20.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.911+0000 7f1fc9600640 1 --2- 172.21.15.28:0/2283417501 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1fa806ac00 0x7f1fa806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:20.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.911+0000 7f1fc9600640 1 -- 172.21.15.28:0/2283417501 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1fb4088a10 con 0x7f1fcc0cafb0 2024-09-13T16:48:20.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.911+0000 7f1fd1000640 1 --2- 172.21.15.28:0/2283417501 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1fa806ac00 0x7f1fa806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:20.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.912+0000 7f1fd1000640 1 --2- 172.21.15.28:0/2283417501 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1fa806ac00 0x7f1fa806d0c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc0011e0 tx=0x7f1fbc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:20.916+0000 7f1fc9600640 1 -- 172.21.15.28:0/2283417501 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1fb4053c10 con 0x7f1fcc0cafb0 2024-09-13T16:48:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.095+0000 7f1fd3a00640 1 -- 172.21.15.28:0/2283417501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1fcc143a30 con 0x7f1fcc0cafb0 2024-09-13T16:48:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.096+0000 7f1fc9600640 1 -- 172.21.15.28:0/2283417501 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1fb4017020 con 0x7f1fcc0cafb0 2024-09-13T16:48:21.097 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T16:48:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.098+0000 7f1fd3a00640 1 -- 172.21.15.28:0/2283417501 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1fa806ac00 msgr2=0x7f1fa806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.098+0000 7f1fd3a00640 1 --2- 172.21.15.28:0/2283417501 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1fa806ac00 0x7f1fa806d0c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc0011e0 tx=0x7f1fbc009040 comp rx=0 tx=0).stop 2024-09-13T16:48:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.098+0000 7f1fd3a00640 1 -- 172.21.15.28:0/2283417501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fcc0cafb0 msgr2=0x7f1fcc14c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.098+0000 7f1fd3a00640 1 --2- 172.21.15.28:0/2283417501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fcc0cafb0 0x7f1fcc14c040 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1fb400c500 tx=0x7f1fb400c9c0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.098+0000 7f1fd3a00640 1 -- 172.21.15.28:0/2283417501 shutdown_connections 2024-09-13T16:48:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.098+0000 7f1fd3a00640 1 --2- 172.21.15.28:0/2283417501 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f1fa806ac00 0x7f1fa806d0c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.098+0000 7f1fd3a00640 1 --2- 172.21.15.28:0/2283417501 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1fcc150b30 0x7f1fcc1de3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.098+0000 7f1fd3a00640 1 --2- 172.21.15.28:0/2283417501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fcc0cafb0 0x7f1fcc14c040 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.098+0000 7f1fd3a00640 1 --2- 172.21.15.28:0/2283417501 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1fcc0c83d0 0x7f1fcc146c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.098+0000 7f1fd3a00640 1 -- 172.21.15.28:0/2283417501 >> 172.21.15.28:0/2283417501 conn(0x7f1fcc0ca2f0 msgr2=0x7f1fcc0c6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.098+0000 7f1fd3a00640 1 -- 172.21.15.28:0/2283417501 shutdown_connections 2024-09-13T16:48:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.098+0000 7f1fd3a00640 1 -- 172.21.15.28:0/2283417501 wait complete. 2024-09-13T16:48:21.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:48:21.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.273+0000 7f8e30a00640 1 Processor -- start 2024-09-13T16:48:21.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.273+0000 7f8e30a00640 1 -- start start 2024-09-13T16:48:21.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.274+0000 7f8e30a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e2c0c9990 0x7f8e2c1594e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:21.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.274+0000 7f8e30a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e2c0b4010 0x7f8e2c0c62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:21.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.274+0000 7f8e30a00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e2c159050 con 0x7f8e2c0c4810 2024-09-13T16:48:21.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.274+0000 7f8e30a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e2c0c7ea0 con 0x7f8e2c0c9990 2024-09-13T16:48:21.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.274+0000 7f8e30a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c0c9e30 con 0x7f8e2c0b4010 2024-09-13T16:48:21.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.274+0000 7f8e22a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e2c0c9990 0x7f8e2c1594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.274+0000 7f8e22a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e2c0c9990 0x7f8e2c1594e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39114/0 (socket says 172.21.15.28:39114) 2024-09-13T16:48:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.274+0000 7f8e22a00640 1 -- 172.21.15.28:0/4087300462 learned_addr learned my addr 172.21.15.28:0/4087300462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.274+0000 7f8e2be00640 1 --2- 172.21.15.28:0/4087300462 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e2c0b4010 0x7f8e2c0c62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.275+0000 7f8e22a00640 1 -- 172.21.15.28:0/4087300462 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e2c0b4010 msgr2=0x7f8e2c0c62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.275+0000 7f8e22a00640 1 --2- 172.21.15.28:0/4087300462 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e2c0b4010 0x7f8e2c0c62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.275+0000 7f8e22a00640 1 -- 172.21.15.28:0/4087300462 >> v1:172.21.15.191:6789/0 conn(0x7f8e2c0c4810 legacy=0x7f8e2c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:21.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.275+0000 7f8e22a00640 1 -- 172.21.15.28:0/4087300462 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2c0ca0d0 con 0x7f8e2c0c9990 2024-09-13T16:48:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.276+0000 7f8e22a00640 1 --2- 172.21.15.28:0/4087300462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e2c0c9990 0x7f8e2c1594e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8e14009870 tx=0x7f8e1402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87cf62946166d347 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.276+0000 7f8e2aa00640 1 -- 172.21.15.28:0/4087300462 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e14032440 con 0x7f8e2c0c9990 2024-09-13T16:48:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.276+0000 7f8e2aa00640 1 -- 172.21.15.28:0/4087300462 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e140325a0 con 0x7f8e2c0c9990 2024-09-13T16:48:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.276+0000 7f8e2aa00640 1 -- 172.21.15.28:0/4087300462 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e14032930 con 0x7f8e2c0c9990 2024-09-13T16:48:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.276+0000 7f8e30a00640 1 -- 172.21.15.28:0/4087300462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e2c0c9990 msgr2=0x7f8e2c1594e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.276+0000 7f8e30a00640 1 --2- 172.21.15.28:0/4087300462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e2c0c9990 0x7f8e2c1594e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8e14009870 tx=0x7f8e1402a270 comp rx=0 tx=0).stop 2024-09-13T16:48:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.277+0000 7f8e30a00640 1 -- 172.21.15.28:0/4087300462 shutdown_connections 2024-09-13T16:48:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.277+0000 7f8e30a00640 1 --2- 172.21.15.28:0/4087300462 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e2c0b4010 0x7f8e2c0c62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.277+0000 7f8e30a00640 1 --2- 172.21.15.28:0/4087300462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e2c0c9990 0x7f8e2c1594e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.277+0000 7f8e30a00640 1 -- 172.21.15.28:0/4087300462 >> 172.21.15.28:0/4087300462 conn(0x7f8e2c06c810 msgr2=0x7f8e2c06a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.277+0000 7f8e30a00640 1 -- 172.21.15.28:0/4087300462 shutdown_connections 2024-09-13T16:48:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.277+0000 7f8e30a00640 1 -- 172.21.15.28:0/4087300462 wait complete. 2024-09-13T16:48:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.278+0000 7f8e30a00640 1 Processor -- start 2024-09-13T16:48:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.278+0000 7f8e30a00640 1 -- start start 2024-09-13T16:48:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.279+0000 7f8e30a00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e2c0b4010 0x7f8e2c0cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.279+0000 7f8e30a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e2c0c4810 0x7f8e2c153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.279+0000 7f8e30a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e2c0c9990 0x7f8e2c0cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.279+0000 7f8e30a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c1ef6a0 con 0x7f8e2c0c9990 2024-09-13T16:48:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.279+0000 7f8e30a00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c0cd8b0 con 0x7f8e2c0b4010 2024-09-13T16:48:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.279+0000 7f8e30a00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c155360 con 0x7f8e2c0c4810 2024-09-13T16:48:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.279+0000 7f8e2b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e2c0b4010 0x7f8e2c0cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.280+0000 7f8e22a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e2c0c4810 0x7f8e2c153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.280+0000 7f8e2b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e2c0b4010 0x7f8e2c0cc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:38624/0 (socket says 172.21.15.28:38624) 2024-09-13T16:48:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.280+0000 7f8e22a00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e2c0c4810 0x7f8e2c153570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:53502/0 (socket says 172.21.15.28:53502) 2024-09-13T16:48:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.280+0000 7f8e2b400640 1 -- 172.21.15.28:0/923749334 learned_addr learned my addr 172.21.15.28:0/923749334 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.280+0000 7f8e2be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e2c0c9990 0x7f8e2c0cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.280+0000 7f8e2b400640 1 -- 172.21.15.28:0/923749334 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e2c0c4810 msgr2=0x7f8e2c153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.280+0000 7f8e2b400640 1 --2- 172.21.15.28:0/923749334 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e2c0c4810 0x7f8e2c153570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.280+0000 7f8e2b400640 1 -- 172.21.15.28:0/923749334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e2c0c9990 msgr2=0x7f8e2c0cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.280+0000 7f8e2b400640 1 --2- 172.21.15.28:0/923749334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e2c0c9990 0x7f8e2c0cdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.280+0000 7f8e2b400640 1 -- 172.21.15.28:0/923749334 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e140094d0 con 0x7f8e2c0b4010 2024-09-13T16:48:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.280+0000 7f8e2be00640 1 --2- 172.21.15.28:0/923749334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e2c0c9990 0x7f8e2c0cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.280+0000 7f8e2b400640 1 --2- 172.21.15.28:0/923749334 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e2c0b4010 0x7f8e2c0cc0c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00c870 tx=0x7f8e1c00cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.280+0000 7f8e23e00640 1 -- 172.21.15.28:0/923749334 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c003b70 con 0x7f8e2c0b4010 2024-09-13T16:48:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.281+0000 7f8e23e00640 1 -- 172.21.15.28:0/923749334 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1c003cf0 con 0x7f8e2c0b4010 2024-09-13T16:48:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.281+0000 7f8e23e00640 1 -- 172.21.15.28:0/923749334 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c00aa90 con 0x7f8e2c0b4010 2024-09-13T16:48:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.281+0000 7f8e30a00640 1 -- 172.21.15.28:0/923749334 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e2c155640 con 0x7f8e2c0b4010 2024-09-13T16:48:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.281+0000 7f8e30a00640 1 -- 172.21.15.28:0/923749334 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e2c155c00 con 0x7f8e2c0b4010 2024-09-13T16:48:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.282+0000 7f8e30a00640 1 -- 172.21.15.28:0/923749334 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e2c0cd6a0 con 0x7f8e2c0b4010 2024-09-13T16:48:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.283+0000 7f8e23e00640 1 -- 172.21.15.28:0/923749334 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8e1c00abf0 con 0x7f8e2c0b4010 2024-09-13T16:48:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.283+0000 7f8e23e00640 1 --2- 172.21.15.28:0/923749334 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8e0806ac00 0x7f8e0806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.284+0000 7f8e23e00640 1 -- 172.21.15.28:0/923749334 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8e1c089190 con 0x7f8e2c0b4010 2024-09-13T16:48:21.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.284+0000 7f8e22a00640 1 --2- 172.21.15.28:0/923749334 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8e0806ac00 0x7f8e0806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:21.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.284+0000 7f8e22a00640 1 --2- 172.21.15.28:0/923749334 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8e0806ac00 0x7f8e0806d0c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8e14034000 tx=0x7f8e1402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:21.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.288+0000 7f8e23e00640 1 -- 172.21.15.28:0/923749334 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8e1c054390 con 0x7f8e2c0b4010 2024-09-13T16:48:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.468+0000 7f8e30a00640 1 -- 172.21.15.28:0/923749334 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8e2c155df0 con 0x7f8e2c0b4010 2024-09-13T16:48:21.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.468+0000 7f8e23e00640 1 -- 172.21.15.28:0/923749334 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8e1c0579b0 con 0x7f8e2c0b4010 2024-09-13T16:48:21.470 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T16:48:21.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.471+0000 7f8e30a00640 1 -- 172.21.15.28:0/923749334 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8e0806ac00 msgr2=0x7f8e0806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.471+0000 7f8e30a00640 1 --2- 172.21.15.28:0/923749334 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8e0806ac00 0x7f8e0806d0c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8e14034000 tx=0x7f8e1402e000 comp rx=0 tx=0).stop 2024-09-13T16:48:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.471+0000 7f8e30a00640 1 -- 172.21.15.28:0/923749334 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e2c0b4010 msgr2=0x7f8e2c0cc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.471+0000 7f8e30a00640 1 --2- 172.21.15.28:0/923749334 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e2c0b4010 0x7f8e2c0cc0c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00c870 tx=0x7f8e1c00cd30 comp rx=0 tx=0).stop 2024-09-13T16:48:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.471+0000 7f8e30a00640 1 -- 172.21.15.28:0/923749334 shutdown_connections 2024-09-13T16:48:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.471+0000 7f8e30a00640 1 --2- 172.21.15.28:0/923749334 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f8e0806ac00 0x7f8e0806d0c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.471+0000 7f8e30a00640 1 --2- 172.21.15.28:0/923749334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e2c0c9990 0x7f8e2c0cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.471+0000 7f8e30a00640 1 --2- 172.21.15.28:0/923749334 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e2c0c4810 0x7f8e2c153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.471+0000 7f8e30a00640 1 --2- 172.21.15.28:0/923749334 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e2c0b4010 0x7f8e2c0cc0c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.471+0000 7f8e30a00640 1 -- 172.21.15.28:0/923749334 >> 172.21.15.28:0/923749334 conn(0x7f8e2c06c810 msgr2=0x7f8e2c0c1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.471+0000 7f8e30a00640 1 -- 172.21.15.28:0/923749334 shutdown_connections 2024-09-13T16:48:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.471+0000 7f8e30a00640 1 -- 172.21.15.28:0/923749334 wait complete. 2024-09-13T16:48:21.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:48:21.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.646+0000 7f5426e00640 1 Processor -- start 2024-09-13T16:48:21.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.646+0000 7f5426e00640 1 -- start start 2024-09-13T16:48:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.647+0000 7f5426e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f542006dcb0 0x7f54200c5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.647+0000 7f5426e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54200c1d80 con 0x7f542015a060 2024-09-13T16:48:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.647+0000 7f5426e00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542015a920 con 0x7f542015a440 2024-09-13T16:48:21.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.647+0000 7f5426e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f542015ac00 con 0x7f542006dcb0 2024-09-13T16:48:21.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.647+0000 7f541fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f542015a060 0x7f542006d5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52692/0 (socket says 172.21.15.28:52692) 2024-09-13T16:48:21.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.647+0000 7f541fe00640 1 -- 172.21.15.28:0/2220143779 learned_addr learned my addr 172.21.15.28:0/2220143779 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:21.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.647+0000 7f5424e00640 1 --2- 172.21.15.28:0/2220143779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f542006dcb0 0x7f54200c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:21.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.648+0000 7f541f400640 1 -- 172.21.15.28:0/2220143779 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021718385 0 0) 0x7f542015a920 con 0x7f542015a440 2024-09-13T16:48:21.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.648+0000 7f541f400640 1 -- 172.21.15.28:0/2220143779 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5404003520 con 0x7f542015a440 2024-09-13T16:48:21.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.648+0000 7f5424e00640 1 -- 172.21.15.28:0/2220143779 >> v1:172.21.15.191:6789/0 conn(0x7f542015a440 legacy=0x7f54201504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:21.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.648+0000 7f5424e00640 1 -- 172.21.15.28:0/2220143779 >> v1:172.21.15.28:6789/0 conn(0x7f542015a060 legacy=0x7f542006d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:21.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.648+0000 7f541f400640 1 -- 172.21.15.28:0/2220143779 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656421963 0 0) 0x7f54200c1d80 con 0x7f542015a060 2024-09-13T16:48:21.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.648+0000 7f5424e00640 1 -- 172.21.15.28:0/2220143779 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54200c2d50 con 0x7f542006dcb0 2024-09-13T16:48:21.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.649+0000 7f5424e00640 1 --2- 172.21.15.28:0/2220143779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f542006dcb0 0x7f54200c5500 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5418009df0 tx=0x7f541802a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=268be7373bc2a5bc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:21.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.649+0000 7f541f400640 1 -- 172.21.15.28:0/2220143779 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5418007740 con 0x7f542006dcb0 2024-09-13T16:48:21.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.649+0000 7f541f400640 1 -- 172.21.15.28:0/2220143779 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541802d050 con 0x7f542006dcb0 2024-09-13T16:48:21.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.649+0000 7f541f400640 1 -- 172.21.15.28:0/2220143779 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5418007aa0 con 0x7f542006dcb0 2024-09-13T16:48:21.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.650+0000 7f5426e00640 1 -- 172.21.15.28:0/2220143779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f542006dcb0 msgr2=0x7f54200c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:21.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.650+0000 7f5426e00640 1 --2- 172.21.15.28:0/2220143779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f542006dcb0 0x7f54200c5500 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5418009df0 tx=0x7f541802a710 comp rx=0 tx=0).stop 2024-09-13T16:48:21.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.650+0000 7f5426e00640 1 -- 172.21.15.28:0/2220143779 shutdown_connections 2024-09-13T16:48:21.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.650+0000 7f5426e00640 1 --2- 172.21.15.28:0/2220143779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f542006dcb0 0x7f54200c5500 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.650+0000 7f5426e00640 1 -- 172.21.15.28:0/2220143779 >> 172.21.15.28:0/2220143779 conn(0x7f54200ca6b0 msgr2=0x7f54200caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:21.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.650+0000 7f5426e00640 1 -- 172.21.15.28:0/2220143779 shutdown_connections 2024-09-13T16:48:21.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.650+0000 7f5426e00640 1 -- 172.21.15.28:0/2220143779 wait complete. 2024-09-13T16:48:21.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.652+0000 7f5426e00640 1 Processor -- start 2024-09-13T16:48:21.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.652+0000 7f5426e00640 1 -- start start 2024-09-13T16:48:21.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.653+0000 7f5426e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f542006dcb0 0x7f5420158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.653+0000 7f5426e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f542015a060 0x7f54200b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.653+0000 7f5426e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f542015a440 0x7f5420070770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.653+0000 7f5426e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54201eec20 con 0x7f542006dcb0 2024-09-13T16:48:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.653+0000 7f5426e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54200702e0 con 0x7f542015a060 2024-09-13T16:48:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.653+0000 7f5426e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54201ef110 con 0x7f542015a440 2024-09-13T16:48:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.653+0000 7f541fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f542006dcb0 0x7f5420158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.653+0000 7f5417e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f542015a060 0x7f54200b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.653+0000 7f5417e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f542015a060 0x7f54200b74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:38628/0 (socket says 172.21.15.28:38628) 2024-09-13T16:48:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.653+0000 7f5417e00640 1 -- 172.21.15.28:0/1726866762 learned_addr learned my addr 172.21.15.28:0/1726866762 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.653+0000 7f5424e00640 1 --2- 172.21.15.28:0/1726866762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f542015a440 0x7f5420070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.653+0000 7f541fe00640 1 -- 172.21.15.28:0/1726866762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f542015a440 msgr2=0x7f5420070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.653+0000 7f541fe00640 1 --2- 172.21.15.28:0/1726866762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f542015a440 0x7f5420070770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.653+0000 7f541fe00640 1 -- 172.21.15.28:0/1726866762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f542015a060 msgr2=0x7f54200b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.653+0000 7f541fe00640 1 --2- 172.21.15.28:0/1726866762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f542015a060 0x7f54200b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.653+0000 7f541fe00640 1 -- 172.21.15.28:0/1726866762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5418009a70 con 0x7f542006dcb0 2024-09-13T16:48:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.653+0000 7f5417e00640 1 --2- 172.21.15.28:0/1726866762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f542015a060 0x7f54200b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.654+0000 7f541fe00640 1 --2- 172.21.15.28:0/1726866762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f542006dcb0 0x7f5420158cf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f541000c410 tx=0x7f541000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.654+0000 7f541cc00640 1 -- 172.21.15.28:0/1726866762 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54100074e0 con 0x7f542006dcb0 2024-09-13T16:48:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.654+0000 7f541cc00640 1 -- 172.21.15.28:0/1726866762 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5410007640 con 0x7f542006dcb0 2024-09-13T16:48:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.654+0000 7f541cc00640 1 -- 172.21.15.28:0/1726866762 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5410004a20 con 0x7f542006dcb0 2024-09-13T16:48:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.654+0000 7f5426e00640 1 -- 172.21.15.28:0/1726866762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54201e2660 con 0x7f542006dcb0 2024-09-13T16:48:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.654+0000 7f5426e00640 1 -- 172.21.15.28:0/1726866762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54201eee30 con 0x7f542006dcb0 2024-09-13T16:48:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.655+0000 7f5426e00640 1 -- 172.21.15.28:0/1726866762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54200c62a0 con 0x7f542006dcb0 2024-09-13T16:48:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.656+0000 7f541cc00640 1 -- 172.21.15.28:0/1726866762 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5410004b80 con 0x7f542006dcb0 2024-09-13T16:48:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.657+0000 7f541cc00640 1 --2- 172.21.15.28:0/1726866762 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f53fc06ac00 0x7f53fc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:21.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.657+0000 7f541cc00640 1 -- 172.21.15.28:0/1726866762 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5410088f70 con 0x7f542006dcb0 2024-09-13T16:48:21.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.657+0000 7f5417e00640 1 --2- 172.21.15.28:0/1726866762 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f53fc06ac00 0x7f53fc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:21.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.658+0000 7f5417e00640 1 --2- 172.21.15.28:0/1726866762 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f53fc06ac00 0x7f53fc06d0c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5420159b50 tx=0x7f5408009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.661+0000 7f541cc00640 1 -- 172.21.15.28:0/1726866762 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f54100bc9c0 con 0x7f542006dcb0 2024-09-13T16:48:21.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.839+0000 7f5426e00640 1 -- 172.21.15.28:0/1726866762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5420154220 con 0x7f542006dcb0 2024-09-13T16:48:21.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.840+0000 7f541cc00640 1 -- 172.21.15.28:0/1726866762 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f54100542d0 con 0x7f542006dcb0 2024-09-13T16:48:21.841 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T16:48:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.842+0000 7f5426e00640 1 -- 172.21.15.28:0/1726866762 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f53fc06ac00 msgr2=0x7f53fc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.842+0000 7f5426e00640 1 --2- 172.21.15.28:0/1726866762 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f53fc06ac00 0x7f53fc06d0c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5420159b50 tx=0x7f5408009040 comp rx=0 tx=0).stop 2024-09-13T16:48:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.842+0000 7f5426e00640 1 -- 172.21.15.28:0/1726866762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f542006dcb0 msgr2=0x7f5420158cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.842+0000 7f5426e00640 1 --2- 172.21.15.28:0/1726866762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f542006dcb0 0x7f5420158cf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f541000c410 tx=0x7f541000c8d0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.842+0000 7f5426e00640 1 -- 172.21.15.28:0/1726866762 shutdown_connections 2024-09-13T16:48:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.842+0000 7f5426e00640 1 --2- 172.21.15.28:0/1726866762 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f53fc06ac00 0x7f53fc06d0c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.842+0000 7f5426e00640 1 --2- 172.21.15.28:0/1726866762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f542015a440 0x7f5420070770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.842+0000 7f5426e00640 1 --2- 172.21.15.28:0/1726866762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f542015a060 0x7f54200b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.842+0000 7f5426e00640 1 --2- 172.21.15.28:0/1726866762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f542006dcb0 0x7f5420158cf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.842+0000 7f5426e00640 1 -- 172.21.15.28:0/1726866762 >> 172.21.15.28:0/1726866762 conn(0x7f54200ca6b0 msgr2=0x7f542014f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.842+0000 7f5426e00640 1 -- 172.21.15.28:0/1726866762 shutdown_connections 2024-09-13T16:48:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:21.842+0000 7f5426e00640 1 -- 172.21.15.28:0/1726866762 wait complete. 2024-09-13T16:48:21.859 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:48:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.013+0000 7f36ffe00640 1 Processor -- start 2024-09-13T16:48:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.014+0000 7f36ffe00640 1 -- start start 2024-09-13T16:48:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.015+0000 7f36ffe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3700071d40 0x7f37000c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.015+0000 7f36ffe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37000c33c0 con 0x7f3700071960 2024-09-13T16:48:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.015+0000 7f36ffe00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37000c4f60 con 0x7f37000c5930 2024-09-13T16:48:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.015+0000 7f36ffe00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3700155c10 con 0x7f3700071d40 2024-09-13T16:48:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.015+0000 7f36f7e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3700071d40 0x7f37000c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.015+0000 7f36fea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3700071960 0x7f37000c17f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52704/0 (socket says 172.21.15.28:52704) 2024-09-13T16:48:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.015+0000 7f36ff400640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f37000c5930 0x7f370014f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:41386/0 (socket says 172.21.15.28:41386) 2024-09-13T16:48:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.015+0000 7f36fea00640 1 -- 172.21.15.28:0/1191666890 learned_addr learned my addr 172.21.15.28:0/1191666890 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.015+0000 7f36f7e00640 1 -- 172.21.15.28:0/1191666890 >> v1:172.21.15.191:6789/0 conn(0x7f37000c5930 legacy=0x7f370014f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.015+0000 7f36f7e00640 1 -- 172.21.15.28:0/1191666890 >> v1:172.21.15.28:6789/0 conn(0x7f3700071960 legacy=0x7f37000c17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.015+0000 7f36f7e00640 1 -- 172.21.15.28:0/1191666890 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3700155ed0 con 0x7f3700071d40 2024-09-13T16:48:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.017+0000 7f36f7e00640 1 --2- 172.21.15.28:0/1191666890 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3700071d40 0x7f37000c53f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f36e80097d0 tx=0x7f36e802c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=571403dbe9c41183 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.017+0000 7f36fe000640 1 -- 172.21.15.28:0/1191666890 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e8031710 con 0x7f3700071d40 2024-09-13T16:48:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.017+0000 7f36fe000640 1 -- 172.21.15.28:0/1191666890 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e802f050 con 0x7f3700071d40 2024-09-13T16:48:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.017+0000 7f36fe000640 1 -- 172.21.15.28:0/1191666890 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e8031a90 con 0x7f3700071d40 2024-09-13T16:48:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.017+0000 7f36ffe00640 1 -- 172.21.15.28:0/1191666890 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3700071d40 msgr2=0x7f37000c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.017+0000 7f36ffe00640 1 --2- 172.21.15.28:0/1191666890 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3700071d40 0x7f37000c53f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f36e80097d0 tx=0x7f36e802c4a0 comp rx=0 tx=0).stop 2024-09-13T16:48:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.017+0000 7f36ffe00640 1 -- 172.21.15.28:0/1191666890 shutdown_connections 2024-09-13T16:48:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.017+0000 7f36ffe00640 1 --2- 172.21.15.28:0/1191666890 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3700071d40 0x7f37000c53f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:22.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.017+0000 7f36ffe00640 1 -- 172.21.15.28:0/1191666890 >> 172.21.15.28:0/1191666890 conn(0x7f3700069e80 msgr2=0x7f370006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:22.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.018+0000 7f36ffe00640 1 -- 172.21.15.28:0/1191666890 shutdown_connections 2024-09-13T16:48:22.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.018+0000 7f36ffe00640 1 -- 172.21.15.28:0/1191666890 wait complete. 2024-09-13T16:48:22.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.019+0000 7f36ffe00640 1 Processor -- start 2024-09-13T16:48:22.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.019+0000 7f36ffe00640 1 -- start start 2024-09-13T16:48:22.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.020+0000 7f36ffe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3700071960 0x7f3700145690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:22.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.020+0000 7f36ffe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3700071d40 0x7f3700148a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:22.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.020+0000 7f36ffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37000c5930 0x7f37001ee720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:22.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.020+0000 7f36ffe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37000c8eb0 con 0x7f37000c5930 2024-09-13T16:48:22.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.020+0000 7f36ffe00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37001ee2d0 con 0x7f3700071960 2024-09-13T16:48:22.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.020+0000 7f36ffe00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3700145bd0 con 0x7f3700071d40 2024-09-13T16:48:22.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.020+0000 7f36ff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37000c5930 0x7f37001ee720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:22.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.020+0000 7f36ff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37000c5930 0x7f37001ee720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39132/0 (socket says 172.21.15.28:39132) 2024-09-13T16:48:22.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.020+0000 7f36ff400640 1 -- 172.21.15.28:0/2315773280 learned_addr learned my addr 172.21.15.28:0/2315773280 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:22.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.021+0000 7f36f7e00640 1 --2- 172.21.15.28:0/2315773280 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3700071d40 0x7f3700148a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:22.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.021+0000 7f36fea00640 1 --2- 172.21.15.28:0/2315773280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3700071960 0x7f3700145690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:22.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.021+0000 7f36ff400640 1 -- 172.21.15.28:0/2315773280 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3700071d40 msgr2=0x7f3700148a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:22.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.021+0000 7f36ff400640 1 --2- 172.21.15.28:0/2315773280 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3700071d40 0x7f3700148a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:22.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.021+0000 7f36ff400640 1 -- 172.21.15.28:0/2315773280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3700071960 msgr2=0x7f3700145690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:22.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.021+0000 7f36ff400640 1 --2- 172.21.15.28:0/2315773280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3700071960 0x7f3700145690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:22.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.021+0000 7f36ff400640 1 -- 172.21.15.28:0/2315773280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e8009400 con 0x7f37000c5930 2024-09-13T16:48:22.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.021+0000 7f36f7e00640 1 --2- 172.21.15.28:0/2315773280 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3700071d40 0x7f3700148a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:22.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.021+0000 7f36fea00640 1 --2- 172.21.15.28:0/2315773280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3700071960 0x7f3700145690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:22.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.021+0000 7f36ff400640 1 --2- 172.21.15.28:0/2315773280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37000c5930 0x7f37001ee720 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f36f800c730 tx=0x7f36f800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:22.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.021+0000 7f36f6a00640 1 -- 172.21.15.28:0/2315773280 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f800ce40 con 0x7f37000c5930 2024-09-13T16:48:22.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.022+0000 7f36f6a00640 1 -- 172.21.15.28:0/2315773280 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f800a640 con 0x7f37000c5930 2024-09-13T16:48:22.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.022+0000 7f36f6a00640 1 -- 172.21.15.28:0/2315773280 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f800a9f0 con 0x7f37000c5930 2024-09-13T16:48:22.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.022+0000 7f36ffe00640 1 -- 172.21.15.28:0/2315773280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37000c8330 con 0x7f37000c5930 2024-09-13T16:48:22.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.022+0000 7f36ffe00640 1 -- 172.21.15.28:0/2315773280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37000c88f0 con 0x7f37000c5930 2024-09-13T16:48:22.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.023+0000 7f36ffe00640 1 -- 172.21.15.28:0/2315773280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37001ee440 con 0x7f37000c5930 2024-09-13T16:48:22.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.024+0000 7f36f6a00640 1 -- 172.21.15.28:0/2315773280 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f36f800ab50 con 0x7f37000c5930 2024-09-13T16:48:22.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.024+0000 7f36f6a00640 1 --2- 172.21.15.28:0/2315773280 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f36dc06ac00 0x7f36dc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:22.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.024+0000 7f36f6a00640 1 -- 172.21.15.28:0/2315773280 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f36f8088af0 con 0x7f37000c5930 2024-09-13T16:48:22.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.025+0000 7f36fea00640 1 --2- 172.21.15.28:0/2315773280 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f36dc06ac00 0x7f36dc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:22.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.025+0000 7f36fea00640 1 --2- 172.21.15.28:0/2315773280 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f36dc06ac00 0x7f36dc06d0c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f36f0005880 tx=0x7f36f0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:22.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.029+0000 7f36f6a00640 1 -- 172.21.15.28:0/2315773280 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f37001ee440 con 0x7f37000c5930 2024-09-13T16:48:22.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.220+0000 7f36ffe00640 1 -- 172.21.15.28:0/2315773280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f37001ee440 con 0x7f37000c5930 2024-09-13T16:48:22.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.221+0000 7f36f6a00640 1 -- 172.21.15.28:0/2315773280 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f37001ee440 con 0x7f37000c5930 2024-09-13T16:48:22.222 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T16:48:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.223+0000 7f36ffe00640 1 -- 172.21.15.28:0/2315773280 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f36dc06ac00 msgr2=0x7f36dc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:22.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.223+0000 7f36ffe00640 1 --2- 172.21.15.28:0/2315773280 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f36dc06ac00 0x7f36dc06d0c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f36f0005880 tx=0x7f36f0009040 comp rx=0 tx=0).stop 2024-09-13T16:48:22.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.223+0000 7f36ffe00640 1 -- 172.21.15.28:0/2315773280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37000c5930 msgr2=0x7f37001ee720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:22.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.223+0000 7f36ffe00640 1 --2- 172.21.15.28:0/2315773280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37000c5930 0x7f37001ee720 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f36f800c730 tx=0x7f36f800cbf0 comp rx=0 tx=0).stop 2024-09-13T16:48:22.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.223+0000 7f36ffe00640 1 -- 172.21.15.28:0/2315773280 shutdown_connections 2024-09-13T16:48:22.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.223+0000 7f36ffe00640 1 --2- 172.21.15.28:0/2315773280 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f36dc06ac00 0x7f36dc06d0c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:22.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.223+0000 7f36ffe00640 1 --2- 172.21.15.28:0/2315773280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37000c5930 0x7f37001ee720 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:22.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.223+0000 7f36ffe00640 1 --2- 172.21.15.28:0/2315773280 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3700071d40 0x7f3700148a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:22.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.223+0000 7f36ffe00640 1 --2- 172.21.15.28:0/2315773280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3700071960 0x7f3700145690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:22.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.223+0000 7f36ffe00640 1 -- 172.21.15.28:0/2315773280 >> 172.21.15.28:0/2315773280 conn(0x7f3700069e80 msgr2=0x7f3700154810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:22.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.223+0000 7f36ffe00640 1 -- 172.21.15.28:0/2315773280 shutdown_connections 2024-09-13T16:48:22.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.223+0000 7f36ffe00640 1 -- 172.21.15.28:0/2315773280 wait complete. 2024-09-13T16:48:22.240 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T16:48:22.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:48:22.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T16:48:22.329 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T16:48:22.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:48:22.329 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:48:22.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:48:22.389 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T16:48:22.425 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:48:22.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:48:22.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:48:22.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:48:22.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-09-13T16:48:22.800 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-13T16:48:22.809 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T16:48:22.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:48:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.966+0000 7f799b200640 1 Processor -- start 2024-09-13T16:48:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.966+0000 7f799b200640 1 -- start start 2024-09-13T16:48:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.967+0000 7f799b200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7994150ca0 0x7f79941568e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.967+0000 7f799b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7994151080 con 0x7f79940c2250 2024-09-13T16:48:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.967+0000 7f799b200640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799406a990 con 0x7f79940c4d60 2024-09-13T16:48:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.967+0000 7f799b200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79940c2670 con 0x7f7994150ca0 2024-09-13T16:48:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.968+0000 7f798be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f79940c2250 0x7f799415a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52714/0 (socket says 172.21.15.28:52714) 2024-09-13T16:48:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.968+0000 7f798be00640 1 -- 172.21.15.28:0/978272270 learned_addr learned my addr 172.21.15.28:0/978272270 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.968+0000 7f7993e00640 1 --2- 172.21.15.28:0/978272270 >> v2:172.21.15.191:3300/0 conn(0x7f7994150ca0 0x7f79941568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.968+0000 7f7993400640 1 -- 172.21.15.28:0/978272270 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717060977 0 0) 0x7f7994151080 con 0x7f79940c2250 2024-09-13T16:48:22.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.968+0000 7f7993400640 1 -- 172.21.15.28:0/978272270 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7978003520 con 0x7f79940c2250 2024-09-13T16:48:22.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.968+0000 7f7993e00640 1 -- 172.21.15.28:0/978272270 >> v1:172.21.15.191:6789/0 conn(0x7f79940c4d60 legacy=0x7f7994161220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:22.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.968+0000 7f7993e00640 1 -- 172.21.15.28:0/978272270 >> v1:172.21.15.28:6789/0 conn(0x7f79940c2250 legacy=0x7f799415a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:22.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.968+0000 7f7993e00640 1 -- 172.21.15.28:0/978272270 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79940c2870 con 0x7f7994150ca0 2024-09-13T16:48:22.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.969+0000 7f7993e00640 1 --2- 172.21.15.28:0/978272270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7994150ca0 0x7f79941568e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f79840097a0 tx=0x7f798402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43d96ef3e0a1f914 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:22.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.970+0000 7f7993400640 1 -- 172.21.15.28:0/978272270 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7984032440 con 0x7f7994150ca0 2024-09-13T16:48:22.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.970+0000 7f7993400640 1 -- 172.21.15.28:0/978272270 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79840325a0 con 0x7f7994150ca0 2024-09-13T16:48:22.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.970+0000 7f7993400640 1 -- 172.21.15.28:0/978272270 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7984032910 con 0x7f7994150ca0 2024-09-13T16:48:22.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.970+0000 7f799b200640 1 -- 172.21.15.28:0/978272270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7994150ca0 msgr2=0x7f79941568e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.970+0000 7f799b200640 1 --2- 172.21.15.28:0/978272270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7994150ca0 0x7f79941568e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f79840097a0 tx=0x7f798402a1a0 comp rx=0 tx=0).stop 2024-09-13T16:48:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.970+0000 7f799b200640 1 -- 172.21.15.28:0/978272270 shutdown_connections 2024-09-13T16:48:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.970+0000 7f799b200640 1 --2- 172.21.15.28:0/978272270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7994150ca0 0x7f79941568e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.970+0000 7f799b200640 1 -- 172.21.15.28:0/978272270 >> 172.21.15.28:0/978272270 conn(0x7f7994066040 msgr2=0x7f7994069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.970+0000 7f799b200640 1 -- 172.21.15.28:0/978272270 shutdown_connections 2024-09-13T16:48:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.970+0000 7f799b200640 1 -- 172.21.15.28:0/978272270 wait complete. 2024-09-13T16:48:22.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.972+0000 7f799b200640 1 Processor -- start 2024-09-13T16:48:22.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.972+0000 7f799b200640 1 -- start start 2024-09-13T16:48:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.973+0000 7f799b200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79940c2250 0x7f79941eeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.973+0000 7f799b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79940c4d60 0x7f7994159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.973+0000 7f799b200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7994150ca0 0x7f79940c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.973+0000 7f799b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79940bd2d0 con 0x7f79940c4d60 2024-09-13T16:48:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.973+0000 7f799b200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79940bca30 con 0x7f79940c2250 2024-09-13T16:48:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.973+0000 7f799b200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79940bcd10 con 0x7f7994150ca0 2024-09-13T16:48:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.973+0000 7f798be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79940c4d60 0x7f7994159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.973+0000 7f798be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79940c4d60 0x7f7994159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39142/0 (socket says 172.21.15.28:39142) 2024-09-13T16:48:22.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.973+0000 7f798be00640 1 -- 172.21.15.28:0/571424752 learned_addr learned my addr 172.21.15.28:0/571424752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:22.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.973+0000 7f7993e00640 1 --2- 172.21.15.28:0/571424752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79940c2250 0x7f79941eeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:22.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.973+0000 7f7999200640 1 --2- 172.21.15.28:0/571424752 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7994150ca0 0x7f79940c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.974+0000 7f798be00640 1 -- 172.21.15.28:0/571424752 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7994150ca0 msgr2=0x7f79940c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.974+0000 7f798be00640 1 --2- 172.21.15.28:0/571424752 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7994150ca0 0x7f79940c9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.974+0000 7f798be00640 1 -- 172.21.15.28:0/571424752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79940c2250 msgr2=0x7f79941eeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.974+0000 7f798be00640 1 --2- 172.21.15.28:0/571424752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79940c2250 0x7f79941eeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.974+0000 7f798be00640 1 -- 172.21.15.28:0/571424752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7984009400 con 0x7f79940c4d60 2024-09-13T16:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.974+0000 7f7993e00640 1 --2- 172.21.15.28:0/571424752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79940c2250 0x7f79941eeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.974+0000 7f798be00640 1 --2- 172.21.15.28:0/571424752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79940c4d60 0x7f7994159740 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f797c00c500 tx=0x7f797c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.974+0000 7f7990c00640 1 -- 172.21.15.28:0/571424752 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c005ce0 con 0x7f79940c4d60 2024-09-13T16:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.974+0000 7f7990c00640 1 -- 172.21.15.28:0/571424752 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797c005e40 con 0x7f79940c4d60 2024-09-13T16:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.974+0000 7f7990c00640 1 -- 172.21.15.28:0/571424752 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c007330 con 0x7f79940c4d60 2024-09-13T16:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.974+0000 7f799b200640 1 -- 172.21.15.28:0/571424752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79940bcff0 con 0x7f79940c4d60 2024-09-13T16:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.974+0000 7f799b200640 1 -- 172.21.15.28:0/571424752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79941ee5e0 con 0x7f79940c4d60 2024-09-13T16:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.975+0000 7f799b200640 1 -- 172.21.15.28:0/571424752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79941ee7d0 con 0x7f79940c4d60 2024-09-13T16:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.976+0000 7f7990c00640 1 -- 172.21.15.28:0/571424752 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f797c007550 con 0x7f79940c4d60 2024-09-13T16:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.977+0000 7f7990c00640 1 --2- 172.21.15.28:0/571424752 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f797006ac00 0x7f797006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.977+0000 7f7990c00640 1 -- 172.21.15.28:0/571424752 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f797c088a10 con 0x7f79940c4d60 2024-09-13T16:48:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.977+0000 7f7993e00640 1 --2- 172.21.15.28:0/571424752 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f797006ac00 0x7f797006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.978+0000 7f7993e00640 1 --2- 172.21.15.28:0/571424752 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f797006ac00 0x7f797006d0c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7984034000 tx=0x7f798402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:22.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:22.981+0000 7f7990c00640 1 -- 172.21.15.28:0/571424752 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f797c053cd0 con 0x7f79940c4d60 2024-09-13T16:48:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.155+0000 7f799b200640 1 -- 172.21.15.28:0/571424752 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f79940bd4c0 con 0x7f797006ac00 2024-09-13T16:48:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.164+0000 7f7990c00640 1 -- 172.21.15.28:0/571424752 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289778 (secure 0 0 0) 0x7f79940bd4c0 con 0x7f797006ac00 2024-09-13T16:48:23.169 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:48:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T16:48:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.167+0000 7f799b200640 1 -- 172.21.15.28:0/571424752 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f797006ac00 msgr2=0x7f797006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.167+0000 7f799b200640 1 --2- 172.21.15.28:0/571424752 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f797006ac00 0x7f797006d0c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7984034000 tx=0x7f798402e000 comp rx=0 tx=0).stop 2024-09-13T16:48:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.167+0000 7f799b200640 1 -- 172.21.15.28:0/571424752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79940c4d60 msgr2=0x7f7994159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.167+0000 7f799b200640 1 --2- 172.21.15.28:0/571424752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79940c4d60 0x7f7994159740 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f797c00c500 tx=0x7f797c00c9c0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.168+0000 7f799b200640 1 -- 172.21.15.28:0/571424752 shutdown_connections 2024-09-13T16:48:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.168+0000 7f799b200640 1 --2- 172.21.15.28:0/571424752 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f797006ac00 0x7f797006d0c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.168+0000 7f799b200640 1 --2- 172.21.15.28:0/571424752 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7994150ca0 0x7f79940c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.168+0000 7f799b200640 1 --2- 172.21.15.28:0/571424752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79940c4d60 0x7f7994159740 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.168+0000 7f799b200640 1 --2- 172.21.15.28:0/571424752 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79940c2250 0x7f79941eeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.168+0000 7f799b200640 1 -- 172.21.15.28:0/571424752 >> 172.21.15.28:0/571424752 conn(0x7f7994066040 msgr2=0x7f799414ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.168+0000 7f799b200640 1 -- 172.21.15.28:0/571424752 shutdown_connections 2024-09-13T16:48:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.168+0000 7f799b200640 1 -- 172.21.15.28:0/571424752 wait complete. 2024-09-13T16:48:23.185 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-09-13T16:48:21.652636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":961811,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":121,"num_read_kb":105,"num_write":9777,"num_write_kb":25168803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11239,"ondisk_log_size":11239,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":305020,"kb_used_data":2876,"kb_used_omap":12,"kb_used_meta":302067,"kb_avail":754669700,"statfs":{"total":773094113280,"available":772781772800,"internally_reserved":0,"allocated":2945024,"data_stored":2586015,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12743,"internal_metadata":309317177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":700,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003045"},"pg_stats":[{"pgid":"3.3f","version":"22'179","reported_seq":193,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.519062+0000","last_change":"2024-09-13T16:43:54.987241+0000","last_active":"2024-09-13T16:47:59.519062+0000","last_peered":"2024-09-13T16:47:59.519062+0000","last_clean":"2024-09-13T16:47:59.519062+0000","last_became_active":"2024-09-13T16:43:54.986988+0000","last_became_peered":"2024-09-13T16:43:54.986988+0000","last_unstale":"2024-09-13T16:47:59.519062+0000","last_undegraded":"2024-09-13T16:47:59.519062+0000","last_fullsized":"2024-09-13T16:47:59.519062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"22'134","reported_seq":148,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.454727+0000","last_change":"2024-09-13T16:43:55.001141+0000","last_active":"2024-09-13T16:47:59.454727+0000","last_peered":"2024-09-13T16:47:59.454727+0000","last_clean":"2024-09-13T16:47:59.454727+0000","last_became_active":"2024-09-13T16:43:55.000757+0000","last_became_peered":"2024-09-13T16:43:55.000757+0000","last_unstale":"2024-09-13T16:47:59.454727+0000","last_undegraded":"2024-09-13T16:47:59.454727+0000","last_fullsized":"2024-09-13T16:47:59.454727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"22'205","reported_seq":219,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.510237+0000","last_change":"2024-09-13T16:43:54.986925+0000","last_active":"2024-09-13T16:47:59.510237+0000","last_peered":"2024-09-13T16:47:59.510237+0000","last_clean":"2024-09-13T16:47:59.510237+0000","last_became_active":"2024-09-13T16:43:54.986198+0000","last_became_peered":"2024-09-13T16:43:54.986198+0000","last_unstale":"2024-09-13T16:47:59.510237+0000","last_undegraded":"2024-09-13T16:47:59.510237+0000","last_fullsized":"2024-09-13T16:47:59.510237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:03.127550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"22'125","reported_seq":139,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.433692+0000","last_change":"2024-09-13T16:43:54.992249+0000","last_active":"2024-09-13T16:47:59.433692+0000","last_peered":"2024-09-13T16:47:59.433692+0000","last_clean":"2024-09-13T16:47:59.433692+0000","last_became_active":"2024-09-13T16:43:54.991804+0000","last_became_peered":"2024-09-13T16:43:54.991804+0000","last_unstale":"2024-09-13T16:47:59.433692+0000","last_undegraded":"2024-09-13T16:47:59.433692+0000","last_fullsized":"2024-09-13T16:47:59.433692+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":270336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"22'190","reported_seq":203,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.430620+0000","last_change":"2024-09-13T16:43:54.987564+0000","last_active":"2024-09-13T16:47:59.430620+0000","last_peered":"2024-09-13T16:47:59.430620+0000","last_clean":"2024-09-13T16:47:59.430620+0000","last_became_active":"2024-09-13T16:43:54.987328+0000","last_became_peered":"2024-09-13T16:43:54.987328+0000","last_unstale":"2024-09-13T16:47:59.430620+0000","last_undegraded":"2024-09-13T16:47:59.430620+0000","last_fullsized":"2024-09-13T16:47:59.430620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:36.006730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"22'150","reported_seq":163,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.494433+0000","last_change":"2024-09-13T16:43:55.000019+0000","last_active":"2024-09-13T16:47:59.494433+0000","last_peered":"2024-09-13T16:47:59.494433+0000","last_clean":"2024-09-13T16:47:59.494433+0000","last_became_active":"2024-09-13T16:43:54.999752+0000","last_became_peered":"2024-09-13T16:43:54.999752+0000","last_unstale":"2024-09-13T16:47:59.494433+0000","last_undegraded":"2024-09-13T16:47:59.494433+0000","last_fullsized":"2024-09-13T16:47:59.494433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"22'170","reported_seq":183,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.493499+0000","last_change":"2024-09-13T16:43:54.999533+0000","last_active":"2024-09-13T16:47:59.493499+0000","last_peered":"2024-09-13T16:47:59.493499+0000","last_clean":"2024-09-13T16:47:59.493499+0000","last_became_active":"2024-09-13T16:43:54.999350+0000","last_became_peered":"2024-09-13T16:43:54.999350+0000","last_unstale":"2024-09-13T16:47:59.493499+0000","last_undegraded":"2024-09-13T16:47:59.493499+0000","last_fullsized":"2024-09-13T16:47:59.493499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:29.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"22'156","reported_seq":169,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.511303+0000","last_change":"2024-09-13T16:43:54.998526+0000","last_active":"2024-09-13T16:47:59.511303+0000","last_peered":"2024-09-13T16:47:59.511303+0000","last_clean":"2024-09-13T16:47:59.511303+0000","last_became_active":"2024-09-13T16:43:54.996309+0000","last_became_peered":"2024-09-13T16:43:54.996309+0000","last_unstale":"2024-09-13T16:47:59.511303+0000","last_undegraded":"2024-09-13T16:47:59.511303+0000","last_fullsized":"2024-09-13T16:47:59.511303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"22'144","reported_seq":157,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.520032+0000","last_change":"2024-09-13T16:43:54.993786+0000","last_active":"2024-09-13T16:47:59.520032+0000","last_peered":"2024-09-13T16:47:59.520032+0000","last_clean":"2024-09-13T16:47:59.520032+0000","last_became_active":"2024-09-13T16:43:54.993475+0000","last_became_peered":"2024-09-13T16:43:54.993475+0000","last_unstale":"2024-09-13T16:47:59.520032+0000","last_undegraded":"2024-09-13T16:47:59.520032+0000","last_fullsized":"2024-09-13T16:47:59.520032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"22'174","reported_seq":187,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.513207+0000","last_change":"2024-09-13T16:43:54.986272+0000","last_active":"2024-09-13T16:47:59.513207+0000","last_peered":"2024-09-13T16:47:59.513207+0000","last_clean":"2024-09-13T16:47:59.513207+0000","last_became_active":"2024-09-13T16:43:54.985979+0000","last_became_peered":"2024-09-13T16:43:54.985979+0000","last_unstale":"2024-09-13T16:47:59.513207+0000","last_undegraded":"2024-09-13T16:47:59.513207+0000","last_fullsized":"2024-09-13T16:47:59.513207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"22'168","reported_seq":181,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.448009+0000","last_change":"2024-09-13T16:43:54.986207+0000","last_active":"2024-09-13T16:47:59.448009+0000","last_peered":"2024-09-13T16:47:59.448009+0000","last_clean":"2024-09-13T16:47:59.448009+0000","last_became_active":"2024-09-13T16:43:54.985948+0000","last_became_peered":"2024-09-13T16:43:54.985948+0000","last_unstale":"2024-09-13T16:47:59.448009+0000","last_undegraded":"2024-09-13T16:47:59.448009+0000","last_fullsized":"2024-09-13T16:47:59.448009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"22'176","reported_seq":189,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.438078+0000","last_change":"2024-09-13T16:43:54.986636+0000","last_active":"2024-09-13T16:47:59.438078+0000","last_peered":"2024-09-13T16:47:59.438078+0000","last_clean":"2024-09-13T16:47:59.438078+0000","last_became_active":"2024-09-13T16:43:54.986161+0000","last_became_peered":"2024-09-13T16:43:54.986161+0000","last_unstale":"2024-09-13T16:47:59.438078+0000","last_undegraded":"2024-09-13T16:47:59.438078+0000","last_fullsized":"2024-09-13T16:47:59.438078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:09.340330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"22'160","reported_seq":173,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.511135+0000","last_change":"2024-09-13T16:43:54.997783+0000","last_active":"2024-09-13T16:47:59.511135+0000","last_peered":"2024-09-13T16:47:59.511135+0000","last_clean":"2024-09-13T16:47:59.511135+0000","last_became_active":"2024-09-13T16:43:54.997162+0000","last_became_peered":"2024-09-13T16:43:54.997162+0000","last_unstale":"2024-09-13T16:47:59.511135+0000","last_undegraded":"2024-09-13T16:47:59.511135+0000","last_fullsized":"2024-09-13T16:47:59.511135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"22'173","reported_seq":186,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.497364+0000","last_change":"2024-09-13T16:43:54.995262+0000","last_active":"2024-09-13T16:47:59.497364+0000","last_peered":"2024-09-13T16:47:59.497364+0000","last_clean":"2024-09-13T16:47:59.497364+0000","last_became_active":"2024-09-13T16:43:54.994891+0000","last_became_peered":"2024-09-13T16:43:54.994891+0000","last_unstale":"2024-09-13T16:47:59.497364+0000","last_undegraded":"2024-09-13T16:47:59.497364+0000","last_fullsized":"2024-09-13T16:47:59.497364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"22'185","reported_seq":198,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.492832+0000","last_change":"2024-09-13T16:43:54.981534+0000","last_active":"2024-09-13T16:47:59.492832+0000","last_peered":"2024-09-13T16:47:59.492832+0000","last_clean":"2024-09-13T16:47:59.492832+0000","last_became_active":"2024-09-13T16:43:54.981191+0000","last_became_peered":"2024-09-13T16:43:54.981191+0000","last_unstale":"2024-09-13T16:47:59.492832+0000","last_undegraded":"2024-09-13T16:47:59.492832+0000","last_fullsized":"2024-09-13T16:47:59.492832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"22'151","reported_seq":164,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.506334+0000","last_change":"2024-09-13T16:43:54.994157+0000","last_active":"2024-09-13T16:47:59.506334+0000","last_peered":"2024-09-13T16:47:59.506334+0000","last_clean":"2024-09-13T16:47:59.506334+0000","last_became_active":"2024-09-13T16:43:54.993915+0000","last_became_peered":"2024-09-13T16:43:54.993915+0000","last_unstale":"2024-09-13T16:47:59.506334+0000","last_undegraded":"2024-09-13T16:47:59.506334+0000","last_fullsized":"2024-09-13T16:47:59.506334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"22'167","reported_seq":180,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.509298+0000","last_change":"2024-09-13T16:43:54.987328+0000","last_active":"2024-09-13T16:47:59.509298+0000","last_peered":"2024-09-13T16:47:59.509298+0000","last_clean":"2024-09-13T16:47:59.509298+0000","last_became_active":"2024-09-13T16:43:54.987048+0000","last_became_peered":"2024-09-13T16:43:54.987048+0000","last_unstale":"2024-09-13T16:47:59.509298+0000","last_undegraded":"2024-09-13T16:47:59.509298+0000","last_fullsized":"2024-09-13T16:47:59.509298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"22'132","reported_seq":145,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.442769+0000","last_change":"2024-09-13T16:43:54.985628+0000","last_active":"2024-09-13T16:47:59.442769+0000","last_peered":"2024-09-13T16:47:59.442769+0000","last_clean":"2024-09-13T16:47:59.442769+0000","last_became_active":"2024-09-13T16:43:54.985378+0000","last_became_peered":"2024-09-13T16:43:54.985378+0000","last_unstale":"2024-09-13T16:47:59.442769+0000","last_undegraded":"2024-09-13T16:47:59.442769+0000","last_fullsized":"2024-09-13T16:47:59.442769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"22'150","reported_seq":163,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.510652+0000","last_change":"2024-09-13T16:43:54.996712+0000","last_active":"2024-09-13T16:47:59.510652+0000","last_peered":"2024-09-13T16:47:59.510652+0000","last_clean":"2024-09-13T16:47:59.510652+0000","last_became_active":"2024-09-13T16:43:54.996523+0000","last_became_peered":"2024-09-13T16:43:54.996523+0000","last_unstale":"2024-09-13T16:47:59.510652+0000","last_undegraded":"2024-09-13T16:47:59.510652+0000","last_fullsized":"2024-09-13T16:47:59.510652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"22'159","reported_seq":172,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.501849+0000","last_change":"2024-09-13T16:43:54.984879+0000","last_active":"2024-09-13T16:47:59.501849+0000","last_peered":"2024-09-13T16:47:59.501849+0000","last_clean":"2024-09-13T16:47:59.501849+0000","last_became_active":"2024-09-13T16:43:54.984595+0000","last_became_peered":"2024-09-13T16:43:54.984595+0000","last_unstale":"2024-09-13T16:47:59.501849+0000","last_undegraded":"2024-09-13T16:47:59.501849+0000","last_fullsized":"2024-09-13T16:47:59.501849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"22'156","reported_seq":169,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.509526+0000","last_change":"2024-09-13T16:43:54.996457+0000","last_active":"2024-09-13T16:47:59.509526+0000","last_peered":"2024-09-13T16:47:59.509526+0000","last_clean":"2024-09-13T16:47:59.509526+0000","last_became_active":"2024-09-13T16:43:54.996275+0000","last_became_peered":"2024-09-13T16:43:54.996275+0000","last_unstale":"2024-09-13T16:47:59.509526+0000","last_undegraded":"2024-09-13T16:47:59.509526+0000","last_fullsized":"2024-09-13T16:47:59.509526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"22'189","reported_seq":202,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.497242+0000","last_change":"2024-09-13T16:43:54.985368+0000","last_active":"2024-09-13T16:47:59.497242+0000","last_peered":"2024-09-13T16:47:59.497242+0000","last_clean":"2024-09-13T16:47:59.497242+0000","last_became_active":"2024-09-13T16:43:54.985130+0000","last_became_peered":"2024-09-13T16:43:54.985130+0000","last_unstale":"2024-09-13T16:47:59.497242+0000","last_undegraded":"2024-09-13T16:47:59.497242+0000","last_fullsized":"2024-09-13T16:47:59.497242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"22'150","reported_seq":163,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.439460+0000","last_change":"2024-09-13T16:43:54.991862+0000","last_active":"2024-09-13T16:47:59.439460+0000","last_peered":"2024-09-13T16:47:59.439460+0000","last_clean":"2024-09-13T16:47:59.439460+0000","last_became_active":"2024-09-13T16:43:54.991554+0000","last_became_peered":"2024-09-13T16:43:54.991554+0000","last_unstale":"2024-09-13T16:47:59.439460+0000","last_undegraded":"2024-09-13T16:47:59.439460+0000","last_fullsized":"2024-09-13T16:47:59.439460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"22'156","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.506350+0000","last_change":"2024-09-13T16:44:59.221091+0000","last_active":"2024-09-13T16:47:59.506350+0000","last_peered":"2024-09-13T16:47:59.506350+0000","last_clean":"2024-09-13T16:47:59.506350+0000","last_became_active":"2024-09-13T16:44:59.220762+0000","last_became_peered":"2024-09-13T16:44:59.220762+0000","last_unstale":"2024-09-13T16:47:59.506350+0000","last_undegraded":"2024-09-13T16:47:59.506350+0000","last_fullsized":"2024-09-13T16:47:59.506350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:39.907088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"22'168","reported_seq":182,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.440760+0000","last_change":"2024-09-13T16:43:54.991366+0000","last_active":"2024-09-13T16:47:59.440760+0000","last_peered":"2024-09-13T16:47:59.440760+0000","last_clean":"2024-09-13T16:47:59.440760+0000","last_became_active":"2024-09-13T16:43:54.991127+0000","last_became_peered":"2024-09-13T16:43:54.991127+0000","last_unstale":"2024-09-13T16:47:59.440760+0000","last_undegraded":"2024-09-13T16:47:59.440760+0000","last_fullsized":"2024-09-13T16:47:59.440760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"22'165","reported_seq":179,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.519235+0000","last_change":"2024-09-13T16:43:54.985615+0000","last_active":"2024-09-13T16:47:59.519235+0000","last_peered":"2024-09-13T16:47:59.519235+0000","last_clean":"2024-09-13T16:47:59.519235+0000","last_became_active":"2024-09-13T16:43:54.985299+0000","last_became_peered":"2024-09-13T16:43:54.985299+0000","last_unstale":"2024-09-13T16:47:59.519235+0000","last_undegraded":"2024-09-13T16:47:59.519235+0000","last_fullsized":"2024-09-13T16:47:59.519235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"22'141","reported_seq":155,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.515984+0000","last_change":"2024-09-13T16:43:54.993325+0000","last_active":"2024-09-13T16:47:59.515984+0000","last_peered":"2024-09-13T16:47:59.515984+0000","last_clean":"2024-09-13T16:47:59.515984+0000","last_became_active":"2024-09-13T16:43:54.993009+0000","last_became_peered":"2024-09-13T16:43:54.993009+0000","last_unstale":"2024-09-13T16:47:59.515984+0000","last_undegraded":"2024-09-13T16:47:59.515984+0000","last_fullsized":"2024-09-13T16:47:59.515984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"22'184","reported_seq":198,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.508028+0000","last_change":"2024-09-13T16:43:54.998229+0000","last_active":"2024-09-13T16:47:59.508028+0000","last_peered":"2024-09-13T16:47:59.508028+0000","last_clean":"2024-09-13T16:47:59.508028+0000","last_became_active":"2024-09-13T16:43:54.995774+0000","last_became_peered":"2024-09-13T16:43:54.995774+0000","last_unstale":"2024-09-13T16:47:59.508028+0000","last_undegraded":"2024-09-13T16:47:59.508028+0000","last_fullsized":"2024-09-13T16:47:59.508028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"22'165","reported_seq":179,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.514352+0000","last_change":"2024-09-13T16:43:55.002126+0000","last_active":"2024-09-13T16:47:59.514352+0000","last_peered":"2024-09-13T16:47:59.514352+0000","last_clean":"2024-09-13T16:47:59.514352+0000","last_became_active":"2024-09-13T16:43:55.001713+0000","last_became_peered":"2024-09-13T16:43:55.001713+0000","last_unstale":"2024-09-13T16:47:59.514352+0000","last_undegraded":"2024-09-13T16:47:59.514352+0000","last_fullsized":"2024-09-13T16:47:59.514352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"22'148","reported_seq":162,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.517055+0000","last_change":"2024-09-13T16:43:54.992745+0000","last_active":"2024-09-13T16:47:59.517055+0000","last_peered":"2024-09-13T16:47:59.517055+0000","last_clean":"2024-09-13T16:47:59.517055+0000","last_became_active":"2024-09-13T16:43:54.992508+0000","last_became_peered":"2024-09-13T16:43:54.992508+0000","last_unstale":"2024-09-13T16:47:59.517055+0000","last_undegraded":"2024-09-13T16:47:59.517055+0000","last_fullsized":"2024-09-13T16:47:59.517055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:23.913470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"22'131","reported_seq":145,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.520706+0000","last_change":"2024-09-13T16:43:54.990631+0000","last_active":"2024-09-13T16:47:59.520706+0000","last_peered":"2024-09-13T16:47:59.520706+0000","last_clean":"2024-09-13T16:47:59.520706+0000","last_became_active":"2024-09-13T16:43:54.986277+0000","last_became_peered":"2024-09-13T16:43:54.986277+0000","last_unstale":"2024-09-13T16:47:59.520706+0000","last_undegraded":"2024-09-13T16:47:59.520706+0000","last_fullsized":"2024-09-13T16:47:59.520706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"22'175","reported_seq":189,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.499494+0000","last_change":"2024-09-13T16:43:55.000059+0000","last_active":"2024-09-13T16:47:59.499494+0000","last_peered":"2024-09-13T16:47:59.499494+0000","last_clean":"2024-09-13T16:47:59.499494+0000","last_became_active":"2024-09-13T16:43:54.999778+0000","last_became_peered":"2024-09-13T16:43:54.999778+0000","last_unstale":"2024-09-13T16:47:59.499494+0000","last_undegraded":"2024-09-13T16:47:59.499494+0000","last_fullsized":"2024-09-13T16:47:59.499494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"22'212","reported_seq":225,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.503806+0000","last_change":"2024-09-13T16:43:54.988375+0000","last_active":"2024-09-13T16:47:59.503806+0000","last_peered":"2024-09-13T16:47:59.503806+0000","last_clean":"2024-09-13T16:47:59.503806+0000","last_became_active":"2024-09-13T16:43:54.988131+0000","last_became_peered":"2024-09-13T16:43:54.988131+0000","last_unstale":"2024-09-13T16:47:59.503806+0000","last_undegraded":"2024-09-13T16:47:59.503806+0000","last_fullsized":"2024-09-13T16:47:59.503806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.635755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.219925+0000","last_change":"2024-09-13T16:43:54.555042+0000","last_active":"2024-09-13T16:44:59.219925+0000","last_peered":"2024-09-13T16:44:59.219925+0000","last_clean":"2024-09-13T16:44:59.219925+0000","last_became_active":"2024-09-13T16:43:54.554697+0000","last_became_peered":"2024-09-13T16:43:54.554697+0000","last_unstale":"2024-09-13T16:44:59.219925+0000","last_undegraded":"2024-09-13T16:44:59.219925+0000","last_fullsized":"2024-09-13T16:44:59.219925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"22'165","reported_seq":178,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.517461+0000","last_change":"2024-09-13T16:43:54.998755+0000","last_active":"2024-09-13T16:47:59.517461+0000","last_peered":"2024-09-13T16:47:59.517461+0000","last_clean":"2024-09-13T16:47:59.517461+0000","last_became_active":"2024-09-13T16:43:54.998399+0000","last_became_peered":"2024-09-13T16:43:54.998399+0000","last_unstale":"2024-09-13T16:47:59.517461+0000","last_undegraded":"2024-09-13T16:47:59.517461+0000","last_fullsized":"2024-09-13T16:47:59.517461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.215854+0000","last_change":"2024-09-13T16:43:54.553250+0000","last_active":"2024-09-13T16:44:59.215854+0000","last_peered":"2024-09-13T16:44:59.215854+0000","last_clean":"2024-09-13T16:44:59.215854+0000","last_became_active":"2024-09-13T16:43:54.552977+0000","last_became_peered":"2024-09-13T16:43:54.552977+0000","last_unstale":"2024-09-13T16:44:59.215854+0000","last_undegraded":"2024-09-13T16:44:59.215854+0000","last_fullsized":"2024-09-13T16:44:59.215854+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.219531+0000","last_change":"2024-09-13T16:43:54.558065+0000","last_active":"2024-09-13T16:44:59.219531+0000","last_peered":"2024-09-13T16:44:59.219531+0000","last_clean":"2024-09-13T16:44:59.219531+0000","last_became_active":"2024-09-13T16:43:54.557722+0000","last_became_peered":"2024-09-13T16:43:54.557722+0000","last_unstale":"2024-09-13T16:44:59.219531+0000","last_undegraded":"2024-09-13T16:44:59.219531+0000","last_fullsized":"2024-09-13T16:44:59.219531+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"22'151","reported_seq":164,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.495858+0000","last_change":"2024-09-13T16:43:55.002041+0000","last_active":"2024-09-13T16:47:59.495858+0000","last_peered":"2024-09-13T16:47:59.495858+0000","last_clean":"2024-09-13T16:47:59.495858+0000","last_became_active":"2024-09-13T16:43:55.001512+0000","last_became_peered":"2024-09-13T16:43:55.001512+0000","last_unstale":"2024-09-13T16:47:59.495858+0000","last_undegraded":"2024-09-13T16:47:59.495858+0000","last_fullsized":"2024-09-13T16:47:59.495858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.383328+0000","last_change":"2024-09-13T16:43:54.550389+0000","last_active":"2024-09-13T16:44:59.383328+0000","last_peered":"2024-09-13T16:44:59.383328+0000","last_clean":"2024-09-13T16:44:59.383328+0000","last_became_active":"2024-09-13T16:43:54.550154+0000","last_became_peered":"2024-09-13T16:43:54.550154+0000","last_unstale":"2024-09-13T16:44:59.383328+0000","last_undegraded":"2024-09-13T16:44:59.383328+0000","last_fullsized":"2024-09-13T16:44:59.383328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"22'168","reported_seq":181,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.449760+0000","last_change":"2024-09-13T16:43:54.993377+0000","last_active":"2024-09-13T16:47:59.449760+0000","last_peered":"2024-09-13T16:47:59.449760+0000","last_clean":"2024-09-13T16:47:59.449760+0000","last_became_active":"2024-09-13T16:43:54.993036+0000","last_became_peered":"2024-09-13T16:43:54.993036+0000","last_unstale":"2024-09-13T16:47:59.449760+0000","last_undegraded":"2024-09-13T16:47:59.449760+0000","last_fullsized":"2024-09-13T16:47:59.449760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:13.068438+0000","last_change":"2024-09-13T16:43:54.555528+0000","last_active":"2024-09-13T16:46:13.068438+0000","last_peered":"2024-09-13T16:46:13.068438+0000","last_clean":"2024-09-13T16:46:13.068438+0000","last_became_active":"2024-09-13T16:43:54.555267+0000","last_became_peered":"2024-09-13T16:43:54.555267+0000","last_unstale":"2024-09-13T16:46:13.068438+0000","last_undegraded":"2024-09-13T16:46:13.068438+0000","last_fullsized":"2024-09-13T16:46:13.068438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"22'143","reported_seq":156,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.438299+0000","last_change":"2024-09-13T16:43:54.981706+0000","last_active":"2024-09-13T16:47:59.438299+0000","last_peered":"2024-09-13T16:47:59.438299+0000","last_clean":"2024-09-13T16:47:59.438299+0000","last_became_active":"2024-09-13T16:43:54.981429+0000","last_became_peered":"2024-09-13T16:43:54.981429+0000","last_unstale":"2024-09-13T16:47:59.438299+0000","last_undegraded":"2024-09-13T16:47:59.438299+0000","last_fullsized":"2024-09-13T16:47:59.438299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.382875+0000","last_change":"2024-09-13T16:43:54.554593+0000","last_active":"2024-09-13T16:44:59.382875+0000","last_peered":"2024-09-13T16:44:59.382875+0000","last_clean":"2024-09-13T16:44:59.382875+0000","last_became_active":"2024-09-13T16:43:54.554307+0000","last_became_peered":"2024-09-13T16:43:54.554307+0000","last_unstale":"2024-09-13T16:44:59.382875+0000","last_undegraded":"2024-09-13T16:44:59.382875+0000","last_fullsized":"2024-09-13T16:44:59.382875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"22'195","reported_seq":208,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.497022+0000","last_change":"2024-09-13T16:43:54.993485+0000","last_active":"2024-09-13T16:47:59.497022+0000","last_peered":"2024-09-13T16:47:59.497022+0000","last_clean":"2024-09-13T16:47:59.497022+0000","last_became_active":"2024-09-13T16:43:54.993226+0000","last_became_peered":"2024-09-13T16:43:54.993226+0000","last_unstale":"2024-09-13T16:47:59.497022+0000","last_undegraded":"2024-09-13T16:47:59.497022+0000","last_fullsized":"2024-09-13T16:47:59.497022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'11","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:03.123692+0000","last_change":"2024-09-13T16:43:54.562976+0000","last_active":"2024-09-13T16:48:03.123692+0000","last_peered":"2024-09-13T16:48:03.123692+0000","last_clean":"2024-09-13T16:48:03.123692+0000","last_became_active":"2024-09-13T16:43:54.560480+0000","last_became_peered":"2024-09-13T16:43:54.560480+0000","last_unstale":"2024-09-13T16:48:03.123692+0000","last_undegraded":"2024-09-13T16:48:03.123692+0000","last_fullsized":"2024-09-13T16:48:03.123692+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"22'200","reported_seq":213,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.501250+0000","last_change":"2024-09-13T16:43:54.986459+0000","last_active":"2024-09-13T16:47:59.501250+0000","last_peered":"2024-09-13T16:47:59.501250+0000","last_clean":"2024-09-13T16:47:59.501250+0000","last_became_active":"2024-09-13T16:43:54.986083+0000","last_became_peered":"2024-09-13T16:43:54.986083+0000","last_unstale":"2024-09-13T16:47:59.501250+0000","last_undegraded":"2024-09-13T16:47:59.501250+0000","last_fullsized":"2024-09-13T16:47:59.501250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:51.960459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'165","reported_seq":179,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:14.014679+0000","last_change":"2024-09-13T16:43:54.565929+0000","last_active":"2024-09-13T16:48:14.014679+0000","last_peered":"2024-09-13T16:48:14.014679+0000","last_clean":"2024-09-13T16:48:14.014679+0000","last_became_active":"2024-09-13T16:43:54.565701+0000","last_became_peered":"2024-09-13T16:43:54.565701+0000","last_unstale":"2024-09-13T16:48:14.014679+0000","last_undegraded":"2024-09-13T16:48:14.014679+0000","last_fullsized":"2024-09-13T16:48:14.014679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364449,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":166,"num_write_kb":479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"22'235","reported_seq":248,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.515163+0000","last_change":"2024-09-13T16:43:54.989068+0000","last_active":"2024-09-13T16:47:59.515163+0000","last_peered":"2024-09-13T16:47:59.515163+0000","last_clean":"2024-09-13T16:47:59.515163+0000","last_became_active":"2024-09-13T16:43:54.988779+0000","last_became_peered":"2024-09-13T16:43:54.988779+0000","last_unstale":"2024-09-13T16:47:59.515163+0000","last_undegraded":"2024-09-13T16:47:59.515163+0000","last_fullsized":"2024-09-13T16:47:59.515163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":213,"num_write_kb":581632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.138534+0000","last_change":"2024-09-13T16:43:54.561487+0000","last_active":"2024-09-13T16:44:59.138534+0000","last_peered":"2024-09-13T16:44:59.138534+0000","last_clean":"2024-09-13T16:44:59.138534+0000","last_became_active":"2024-09-13T16:43:54.561077+0000","last_became_peered":"2024-09-13T16:43:54.561077+0000","last_unstale":"2024-09-13T16:44:59.138534+0000","last_undegraded":"2024-09-13T16:44:59.138534+0000","last_fullsized":"2024-09-13T16:44:59.138534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"22'153","reported_seq":166,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.500527+0000","last_change":"2024-09-13T16:43:54.999517+0000","last_active":"2024-09-13T16:47:59.500527+0000","last_peered":"2024-09-13T16:47:59.500527+0000","last_clean":"2024-09-13T16:47:59.500527+0000","last_became_active":"2024-09-13T16:43:54.999283+0000","last_became_peered":"2024-09-13T16:43:54.999283+0000","last_unstale":"2024-09-13T16:47:59.500527+0000","last_undegraded":"2024-09-13T16:47:59.500527+0000","last_fullsized":"2024-09-13T16:47:59.500527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:20.207775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.383950+0000","last_change":"2024-09-13T16:43:54.542871+0000","last_active":"2024-09-13T16:44:59.383950+0000","last_peered":"2024-09-13T16:44:59.383950+0000","last_clean":"2024-09-13T16:44:59.383950+0000","last_became_active":"2024-09-13T16:43:54.542548+0000","last_became_peered":"2024-09-13T16:43:54.542548+0000","last_unstale":"2024-09-13T16:44:59.383950+0000","last_undegraded":"2024-09-13T16:44:59.383950+0000","last_fullsized":"2024-09-13T16:44:59.383950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"22'180","reported_seq":193,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.452606+0000","last_change":"2024-09-13T16:43:54.989168+0000","last_active":"2024-09-13T16:47:59.452606+0000","last_peered":"2024-09-13T16:47:59.452606+0000","last_clean":"2024-09-13T16:47:59.452606+0000","last_became_active":"2024-09-13T16:43:54.988716+0000","last_became_peered":"2024-09-13T16:43:54.988716+0000","last_unstale":"2024-09-13T16:47:59.452606+0000","last_undegraded":"2024-09-13T16:47:59.452606+0000","last_fullsized":"2024-09-13T16:47:59.452606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"20'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.215766+0000","last_change":"2024-09-13T16:43:54.551551+0000","last_active":"2024-09-13T16:44:59.215766+0000","last_peered":"2024-09-13T16:44:59.215766+0000","last_clean":"2024-09-13T16:44:59.215766+0000","last_became_active":"2024-09-13T16:43:54.551293+0000","last_became_peered":"2024-09-13T16:43:54.551293+0000","last_unstale":"2024-09-13T16:44:59.215766+0000","last_undegraded":"2024-09-13T16:44:59.215766+0000","last_fullsized":"2024-09-13T16:44:59.215766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":799,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"22'163","reported_seq":176,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.499067+0000","last_change":"2024-09-13T16:43:54.994516+0000","last_active":"2024-09-13T16:47:59.499067+0000","last_peered":"2024-09-13T16:47:59.499067+0000","last_clean":"2024-09-13T16:47:59.499067+0000","last_became_active":"2024-09-13T16:43:54.994245+0000","last_became_peered":"2024-09-13T16:43:54.994245+0000","last_unstale":"2024-09-13T16:47:59.499067+0000","last_undegraded":"2024-09-13T16:47:59.499067+0000","last_fullsized":"2024-09-13T16:47:59.499067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'18","reported_seq":32,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:14.507552+0000","last_change":"2024-09-13T16:43:54.562232+0000","last_active":"2024-09-13T16:46:14.507552+0000","last_peered":"2024-09-13T16:46:14.507552+0000","last_clean":"2024-09-13T16:46:14.507552+0000","last_became_active":"2024-09-13T16:43:54.561939+0000","last_became_peered":"2024-09-13T16:43:54.561939+0000","last_unstale":"2024-09-13T16:46:14.507552+0000","last_undegraded":"2024-09-13T16:46:14.507552+0000","last_fullsized":"2024-09-13T16:46:14.507552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"22'169","reported_seq":182,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.452403+0000","last_change":"2024-09-13T16:43:54.988439+0000","last_active":"2024-09-13T16:47:59.452403+0000","last_peered":"2024-09-13T16:47:59.452403+0000","last_clean":"2024-09-13T16:47:59.452403+0000","last_became_active":"2024-09-13T16:43:54.988172+0000","last_became_peered":"2024-09-13T16:43:54.988172+0000","last_unstale":"2024-09-13T16:47:59.452403+0000","last_undegraded":"2024-09-13T16:47:59.452403+0000","last_fullsized":"2024-09-13T16:47:59.452403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:58.954977+0000","last_change":"2024-09-13T16:43:54.559372+0000","last_active":"2024-09-13T16:44:58.954977+0000","last_peered":"2024-09-13T16:44:58.954977+0000","last_clean":"2024-09-13T16:44:58.954977+0000","last_became_active":"2024-09-13T16:43:54.559111+0000","last_became_peered":"2024-09-13T16:43:54.559111+0000","last_unstale":"2024-09-13T16:44:58.954977+0000","last_undegraded":"2024-09-13T16:44:58.954977+0000","last_fullsized":"2024-09-13T16:44:58.954977+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:40.399883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'151","reported_seq":164,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.509322+0000","last_change":"2024-09-13T16:43:54.988313+0000","last_active":"2024-09-13T16:47:59.509322+0000","last_peered":"2024-09-13T16:47:59.509322+0000","last_clean":"2024-09-13T16:47:59.509322+0000","last_became_active":"2024-09-13T16:43:54.988059+0000","last_became_peered":"2024-09-13T16:43:54.988059+0000","last_unstale":"2024-09-13T16:47:59.509322+0000","last_undegraded":"2024-09-13T16:47:59.509322+0000","last_fullsized":"2024-09-13T16:47:59.509322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:58.955244+0000","last_change":"2024-09-13T16:43:54.566410+0000","last_active":"2024-09-13T16:44:58.955244+0000","last_peered":"2024-09-13T16:44:58.955244+0000","last_clean":"2024-09-13T16:44:58.955244+0000","last_became_active":"2024-09-13T16:43:54.565861+0000","last_became_peered":"2024-09-13T16:43:54.565861+0000","last_unstale":"2024-09-13T16:44:58.955244+0000","last_undegraded":"2024-09-13T16:44:58.955244+0000","last_fullsized":"2024-09-13T16:44:58.955244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"22'127","reported_seq":138,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.428730+0000","last_change":"2024-09-13T16:44:58.956863+0000","last_active":"2024-09-13T16:47:59.428730+0000","last_peered":"2024-09-13T16:47:59.428730+0000","last_clean":"2024-09-13T16:47:59.428730+0000","last_became_active":"2024-09-13T16:44:58.956557+0000","last_became_peered":"2024-09-13T16:44:58.956557+0000","last_unstale":"2024-09-13T16:47:59.428730+0000","last_undegraded":"2024-09-13T16:47:59.428730+0000","last_fullsized":"2024-09-13T16:47:59.428730+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:10.217411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":278528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:58.957395+0000","last_change":"2024-09-13T16:43:54.559723+0000","last_active":"2024-09-13T16:44:58.957395+0000","last_peered":"2024-09-13T16:44:58.957395+0000","last_clean":"2024-09-13T16:44:58.957395+0000","last_became_active":"2024-09-13T16:43:54.559353+0000","last_became_peered":"2024-09-13T16:43:54.559353+0000","last_unstale":"2024-09-13T16:44:58.957395+0000","last_undegraded":"2024-09-13T16:44:58.957395+0000","last_fullsized":"2024-09-13T16:44:58.957395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"22'203","reported_seq":216,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.498887+0000","last_change":"2024-09-13T16:43:54.992511+0000","last_active":"2024-09-13T16:47:59.498887+0000","last_peered":"2024-09-13T16:47:59.498887+0000","last_clean":"2024-09-13T16:47:59.498887+0000","last_became_active":"2024-09-13T16:43:54.992249+0000","last_became_peered":"2024-09-13T16:43:54.992249+0000","last_unstale":"2024-09-13T16:47:59.498887+0000","last_undegraded":"2024-09-13T16:47:59.498887+0000","last_fullsized":"2024-09-13T16:47:59.498887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"20'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:58.955067+0000","last_change":"2024-09-13T16:43:54.551390+0000","last_active":"2024-09-13T16:44:58.955067+0000","last_peered":"2024-09-13T16:44:58.955067+0000","last_clean":"2024-09-13T16:44:58.955067+0000","last_became_active":"2024-09-13T16:43:54.547074+0000","last_became_peered":"2024-09-13T16:43:54.547074+0000","last_unstale":"2024-09-13T16:44:58.955067+0000","last_undegraded":"2024-09-13T16:44:58.955067+0000","last_fullsized":"2024-09-13T16:44:58.955067+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"22'165","reported_seq":178,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.492614+0000","last_change":"2024-09-13T16:43:54.990458+0000","last_active":"2024-09-13T16:47:59.492614+0000","last_peered":"2024-09-13T16:47:59.492614+0000","last_clean":"2024-09-13T16:47:59.492614+0000","last_became_active":"2024-09-13T16:43:54.990196+0000","last_became_peered":"2024-09-13T16:43:54.990196+0000","last_unstale":"2024-09-13T16:47:59.492614+0000","last_undegraded":"2024-09-13T16:47:59.492614+0000","last_fullsized":"2024-09-13T16:47:59.492614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:54.505478+0000","last_change":"2024-09-13T16:43:51.914639+0000","last_active":"2024-09-13T16:45:54.505478+0000","last_peered":"2024-09-13T16:45:54.505478+0000","last_clean":"2024-09-13T16:45:54.505478+0000","last_became_active":"2024-09-13T16:43:51.913797+0000","last_became_peered":"2024-09-13T16:43:51.913797+0000","last_unstale":"2024-09-13T16:45:54.505478+0000","last_undegraded":"2024-09-13T16:45:54.505478+0000","last_fullsized":"2024-09-13T16:45:54.505478+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_clean_scrub_stamp":"2024-09-13T16:43:50.498116+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:12.730911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:00.501470+0000","last_change":"2024-09-13T16:43:54.564153+0000","last_active":"2024-09-13T16:46:00.501470+0000","last_peered":"2024-09-13T16:46:00.501470+0000","last_clean":"2024-09-13T16:46:00.501470+0000","last_became_active":"2024-09-13T16:43:54.563949+0000","last_became_peered":"2024-09-13T16:43:54.563949+0000","last_unstale":"2024-09-13T16:46:00.501470+0000","last_undegraded":"2024-09-13T16:46:00.501470+0000","last_fullsized":"2024-09-13T16:46:00.501470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"22'207","reported_seq":218,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.437738+0000","last_change":"2024-09-13T16:44:59.387015+0000","last_active":"2024-09-13T16:47:59.437738+0000","last_peered":"2024-09-13T16:47:59.437738+0000","last_clean":"2024-09-13T16:47:59.437738+0000","last_became_active":"2024-09-13T16:44:59.386756+0000","last_became_peered":"2024-09-13T16:44:59.386756+0000","last_unstale":"2024-09-13T16:47:59.437738+0000","last_undegraded":"2024-09-13T16:47:59.437738+0000","last_fullsized":"2024-09-13T16:47:59.437738+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:15.100276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":186,"num_write_kb":507904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:54.505444+0000","last_change":"2024-09-13T16:43:54.564804+0000","last_active":"2024-09-13T16:45:54.505444+0000","last_peered":"2024-09-13T16:45:54.505444+0000","last_clean":"2024-09-13T16:45:54.505444+0000","last_became_active":"2024-09-13T16:43:54.564605+0000","last_became_peered":"2024-09-13T16:43:54.564605+0000","last_unstale":"2024-09-13T16:45:54.505444+0000","last_undegraded":"2024-09-13T16:45:54.505444+0000","last_fullsized":"2024-09-13T16:45:54.505444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"22'187","reported_seq":201,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.514834+0000","last_change":"2024-09-13T16:43:54.997884+0000","last_active":"2024-09-13T16:47:59.514834+0000","last_peered":"2024-09-13T16:47:59.514834+0000","last_clean":"2024-09-13T16:47:59.514834+0000","last_became_active":"2024-09-13T16:43:54.997510+0000","last_became_peered":"2024-09-13T16:43:54.997510+0000","last_unstale":"2024-09-13T16:47:59.514834+0000","last_undegraded":"2024-09-13T16:47:59.514834+0000","last_fullsized":"2024-09-13T16:47:59.514834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'12","reported_seq":32,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:10.123144+0000","last_change":"2024-09-13T16:43:54.566294+0000","last_active":"2024-09-13T16:48:10.123144+0000","last_peered":"2024-09-13T16:48:10.123144+0000","last_clean":"2024-09-13T16:48:10.123144+0000","last_became_active":"2024-09-13T16:43:54.566023+0000","last_became_peered":"2024-09-13T16:43:54.566023+0000","last_unstale":"2024-09-13T16:48:10.123144+0000","last_undegraded":"2024-09-13T16:48:10.123144+0000","last_fullsized":"2024-09-13T16:48:10.123144+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"22'159","reported_seq":173,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.518351+0000","last_change":"2024-09-13T16:43:54.981116+0000","last_active":"2024-09-13T16:47:59.518351+0000","last_peered":"2024-09-13T16:47:59.518351+0000","last_clean":"2024-09-13T16:47:59.518351+0000","last_became_active":"2024-09-13T16:43:54.980766+0000","last_became_peered":"2024-09-13T16:43:54.980766+0000","last_unstale":"2024-09-13T16:47:59.518351+0000","last_undegraded":"2024-09-13T16:47:59.518351+0000","last_fullsized":"2024-09-13T16:47:59.518351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:00.501452+0000","last_change":"2024-09-13T16:43:54.563716+0000","last_active":"2024-09-13T16:46:00.501452+0000","last_peered":"2024-09-13T16:46:00.501452+0000","last_clean":"2024-09-13T16:46:00.501452+0000","last_became_active":"2024-09-13T16:43:54.563522+0000","last_became_peered":"2024-09-13T16:43:54.563522+0000","last_unstale":"2024-09-13T16:46:00.501452+0000","last_undegraded":"2024-09-13T16:46:00.501452+0000","last_fullsized":"2024-09-13T16:46:00.501452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"22'156","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.513433+0000","last_change":"2024-09-13T16:43:54.999051+0000","last_active":"2024-09-13T16:47:59.513433+0000","last_peered":"2024-09-13T16:47:59.513433+0000","last_clean":"2024-09-13T16:47:59.513433+0000","last_became_active":"2024-09-13T16:43:54.998803+0000","last_became_peered":"2024-09-13T16:43:54.998803+0000","last_unstale":"2024-09-13T16:47:59.513433+0000","last_undegraded":"2024-09-13T16:47:59.513433+0000","last_fullsized":"2024-09-13T16:47:59.513433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:59.505325+0000","last_change":"2024-09-13T16:43:54.569945+0000","last_active":"2024-09-13T16:45:59.505325+0000","last_peered":"2024-09-13T16:45:59.505325+0000","last_clean":"2024-09-13T16:45:59.505325+0000","last_became_active":"2024-09-13T16:43:54.568846+0000","last_became_peered":"2024-09-13T16:43:54.568846+0000","last_unstale":"2024-09-13T16:45:59.505325+0000","last_undegraded":"2024-09-13T16:45:59.505325+0000","last_fullsized":"2024-09-13T16:45:59.505325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"22'175","reported_seq":189,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.514055+0000","last_change":"2024-09-13T16:43:54.992082+0000","last_active":"2024-09-13T16:47:59.514055+0000","last_peered":"2024-09-13T16:47:59.514055+0000","last_clean":"2024-09-13T16:47:59.514055+0000","last_became_active":"2024-09-13T16:43:54.991659+0000","last_became_peered":"2024-09-13T16:43:54.991659+0000","last_unstale":"2024-09-13T16:47:59.514055+0000","last_undegraded":"2024-09-13T16:47:59.514055+0000","last_fullsized":"2024-09-13T16:47:59.514055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:59.506045+0000","last_change":"2024-09-13T16:43:54.569111+0000","last_active":"2024-09-13T16:45:59.506045+0000","last_peered":"2024-09-13T16:45:59.506045+0000","last_clean":"2024-09-13T16:45:59.506045+0000","last_became_active":"2024-09-13T16:43:54.568903+0000","last_became_peered":"2024-09-13T16:43:54.568903+0000","last_unstale":"2024-09-13T16:45:59.506045+0000","last_undegraded":"2024-09-13T16:45:59.506045+0000","last_fullsized":"2024-09-13T16:45:59.506045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"22'197","reported_seq":211,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.518940+0000","last_change":"2024-09-13T16:43:54.980100+0000","last_active":"2024-09-13T16:47:59.518940+0000","last_peered":"2024-09-13T16:47:59.518940+0000","last_clean":"2024-09-13T16:47:59.518940+0000","last_became_active":"2024-09-13T16:43:54.979584+0000","last_became_peered":"2024-09-13T16:43:54.979584+0000","last_unstale":"2024-09-13T16:47:59.518940+0000","last_undegraded":"2024-09-13T16:47:59.518940+0000","last_fullsized":"2024-09-13T16:47:59.518940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'20","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:14.014651+0000","last_change":"2024-09-13T16:43:54.560710+0000","last_active":"2024-09-13T16:48:14.014651+0000","last_peered":"2024-09-13T16:48:14.014651+0000","last_clean":"2024-09-13T16:48:14.014651+0000","last_became_active":"2024-09-13T16:43:54.560030+0000","last_became_peered":"2024-09-13T16:43:54.560030+0000","last_unstale":"2024-09-13T16:48:14.014651+0000","last_undegraded":"2024-09-13T16:48:14.014651+0000","last_fullsized":"2024-09-13T16:48:14.014651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'191","reported_seq":205,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.510771+0000","last_change":"2024-09-13T16:43:54.990812+0000","last_active":"2024-09-13T16:47:59.510771+0000","last_peered":"2024-09-13T16:47:59.510771+0000","last_clean":"2024-09-13T16:47:59.510771+0000","last_became_active":"2024-09-13T16:43:54.990440+0000","last_became_peered":"2024-09-13T16:43:54.990440+0000","last_unstale":"2024-09-13T16:47:59.510771+0000","last_undegraded":"2024-09-13T16:47:59.510771+0000","last_fullsized":"2024-09-13T16:47:59.510771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'8","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:03.122078+0000","last_change":"2024-09-13T16:43:54.558439+0000","last_active":"2024-09-13T16:48:03.122078+0000","last_peered":"2024-09-13T16:48:03.122078+0000","last_clean":"2024-09-13T16:48:03.122078+0000","last_became_active":"2024-09-13T16:43:54.558194+0000","last_became_peered":"2024-09-13T16:43:54.558194+0000","last_unstale":"2024-09-13T16:48:03.122078+0000","last_undegraded":"2024-09-13T16:48:03.122078+0000","last_fullsized":"2024-09-13T16:48:03.122078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"22'154","reported_seq":168,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.507374+0000","last_change":"2024-09-13T16:43:54.995534+0000","last_active":"2024-09-13T16:47:59.507374+0000","last_peered":"2024-09-13T16:47:59.507374+0000","last_clean":"2024-09-13T16:47:59.507374+0000","last_became_active":"2024-09-13T16:43:54.995397+0000","last_became_peered":"2024-09-13T16:43:54.995397+0000","last_unstale":"2024-09-13T16:47:59.507374+0000","last_undegraded":"2024-09-13T16:47:59.507374+0000","last_fullsized":"2024-09-13T16:47:59.507374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:19.697247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:04.506432+0000","last_change":"2024-09-13T16:43:54.569085+0000","last_active":"2024-09-13T16:46:04.506432+0000","last_peered":"2024-09-13T16:46:04.506432+0000","last_clean":"2024-09-13T16:46:04.506432+0000","last_became_active":"2024-09-13T16:43:54.568892+0000","last_became_peered":"2024-09-13T16:43:54.568892+0000","last_unstale":"2024-09-13T16:46:04.506432+0000","last_undegraded":"2024-09-13T16:46:04.506432+0000","last_fullsized":"2024-09-13T16:46:04.506432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"22'196","reported_seq":210,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.498566+0000","last_change":"2024-09-13T16:43:54.984120+0000","last_active":"2024-09-13T16:47:59.498566+0000","last_peered":"2024-09-13T16:47:59.498566+0000","last_clean":"2024-09-13T16:47:59.498566+0000","last_became_active":"2024-09-13T16:43:54.983886+0000","last_became_peered":"2024-09-13T16:43:54.983886+0000","last_unstale":"2024-09-13T16:47:59.498566+0000","last_undegraded":"2024-09-13T16:47:59.498566+0000","last_fullsized":"2024-09-13T16:47:59.498566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:04.505849+0000","last_change":"2024-09-13T16:43:54.555092+0000","last_active":"2024-09-13T16:46:04.505849+0000","last_peered":"2024-09-13T16:46:04.505849+0000","last_clean":"2024-09-13T16:46:04.505849+0000","last_became_active":"2024-09-13T16:43:54.554230+0000","last_became_peered":"2024-09-13T16:43:54.554230+0000","last_unstale":"2024-09-13T16:46:04.505849+0000","last_undegraded":"2024-09-13T16:46:04.505849+0000","last_fullsized":"2024-09-13T16:46:04.505849+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"22'169","reported_seq":183,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.513275+0000","last_change":"2024-09-13T16:43:54.984588+0000","last_active":"2024-09-13T16:47:59.513275+0000","last_peered":"2024-09-13T16:47:59.513275+0000","last_clean":"2024-09-13T16:47:59.513275+0000","last_became_active":"2024-09-13T16:43:54.984221+0000","last_became_peered":"2024-09-13T16:43:54.984221+0000","last_unstale":"2024-09-13T16:47:59.513275+0000","last_undegraded":"2024-09-13T16:47:59.513275+0000","last_fullsized":"2024-09-13T16:47:59.513275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:04.505864+0000","last_change":"2024-09-13T16:43:54.562646+0000","last_active":"2024-09-13T16:46:04.505864+0000","last_peered":"2024-09-13T16:46:04.505864+0000","last_clean":"2024-09-13T16:46:04.505864+0000","last_became_active":"2024-09-13T16:43:54.556888+0000","last_became_peered":"2024-09-13T16:43:54.556888+0000","last_unstale":"2024-09-13T16:46:04.505864+0000","last_undegraded":"2024-09-13T16:46:04.505864+0000","last_fullsized":"2024-09-13T16:46:04.505864+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"22'209","reported_seq":222,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.429450+0000","last_change":"2024-09-13T16:43:54.999328+0000","last_active":"2024-09-13T16:47:59.429450+0000","last_peered":"2024-09-13T16:47:59.429450+0000","last_clean":"2024-09-13T16:47:59.429450+0000","last_became_active":"2024-09-13T16:43:54.999101+0000","last_became_peered":"2024-09-13T16:43:54.999101+0000","last_unstale":"2024-09-13T16:47:59.429450+0000","last_undegraded":"2024-09-13T16:47:59.429450+0000","last_fullsized":"2024-09-13T16:47:59.429450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:09.507033+0000","last_change":"2024-09-13T16:43:54.559214+0000","last_active":"2024-09-13T16:46:09.507033+0000","last_peered":"2024-09-13T16:46:09.507033+0000","last_clean":"2024-09-13T16:46:09.507033+0000","last_became_active":"2024-09-13T16:43:54.558233+0000","last_became_peered":"2024-09-13T16:43:54.558233+0000","last_unstale":"2024-09-13T16:46:09.507033+0000","last_undegraded":"2024-09-13T16:46:09.507033+0000","last_fullsized":"2024-09-13T16:46:09.507033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"22'185","reported_seq":199,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.519284+0000","last_change":"2024-09-13T16:43:54.986565+0000","last_active":"2024-09-13T16:47:59.519284+0000","last_peered":"2024-09-13T16:47:59.519284+0000","last_clean":"2024-09-13T16:47:59.519284+0000","last_became_active":"2024-09-13T16:43:54.986084+0000","last_became_peered":"2024-09-13T16:43:54.986084+0000","last_unstale":"2024-09-13T16:47:59.519284+0000","last_undegraded":"2024-09-13T16:47:59.519284+0000","last_fullsized":"2024-09-13T16:47:59.519284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"22'150","reported_seq":163,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.519001+0000","last_change":"2024-09-13T16:43:54.982611+0000","last_active":"2024-09-13T16:47:59.519001+0000","last_peered":"2024-09-13T16:47:59.519001+0000","last_clean":"2024-09-13T16:47:59.519001+0000","last_became_active":"2024-09-13T16:43:54.982338+0000","last_became_peered":"2024-09-13T16:43:54.982338+0000","last_unstale":"2024-09-13T16:47:59.519001+0000","last_undegraded":"2024-09-13T16:47:59.519001+0000","last_fullsized":"2024-09-13T16:47:59.519001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.382688+0000","last_change":"2024-09-13T16:43:54.558799+0000","last_active":"2024-09-13T16:44:59.382688+0000","last_peered":"2024-09-13T16:44:59.382688+0000","last_clean":"2024-09-13T16:44:59.382688+0000","last_became_active":"2024-09-13T16:43:54.558332+0000","last_became_peered":"2024-09-13T16:43:54.558332+0000","last_unstale":"2024-09-13T16:44:59.382688+0000","last_undegraded":"2024-09-13T16:44:59.382688+0000","last_fullsized":"2024-09-13T16:44:59.382688+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:19.201506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'143","reported_seq":154,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.415246+0000","last_change":"2024-09-13T16:44:58.957875+0000","last_active":"2024-09-13T16:47:59.415246+0000","last_peered":"2024-09-13T16:47:59.415246+0000","last_clean":"2024-09-13T16:47:59.415246+0000","last_became_active":"2024-09-13T16:44:58.957568+0000","last_became_peered":"2024-09-13T16:44:58.957568+0000","last_unstale":"2024-09-13T16:47:59.415246+0000","last_undegraded":"2024-09-13T16:47:59.415246+0000","last_fullsized":"2024-09-13T16:47:59.415246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:28.185704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'13","reported_seq":27,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:13.069441+0000","last_change":"2024-09-13T16:43:54.559551+0000","last_active":"2024-09-13T16:46:13.069441+0000","last_peered":"2024-09-13T16:46:13.069441+0000","last_clean":"2024-09-13T16:46:13.069441+0000","last_became_active":"2024-09-13T16:43:54.559290+0000","last_became_peered":"2024-09-13T16:43:54.559290+0000","last_unstale":"2024-09-13T16:46:13.069441+0000","last_undegraded":"2024-09-13T16:46:13.069441+0000","last_fullsized":"2024-09-13T16:46:13.069441+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"22'174","reported_seq":188,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.508073+0000","last_change":"2024-09-13T16:43:55.000783+0000","last_active":"2024-09-13T16:47:59.508073+0000","last_peered":"2024-09-13T16:47:59.508073+0000","last_clean":"2024-09-13T16:47:59.508073+0000","last_became_active":"2024-09-13T16:43:55.000196+0000","last_became_peered":"2024-09-13T16:43:55.000196+0000","last_unstale":"2024-09-13T16:47:59.508073+0000","last_undegraded":"2024-09-13T16:47:59.508073+0000","last_fullsized":"2024-09-13T16:47:59.508073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:54.504808+0000","last_change":"2024-09-13T16:43:54.559886+0000","last_active":"2024-09-13T16:45:54.504808+0000","last_peered":"2024-09-13T16:45:54.504808+0000","last_clean":"2024-09-13T16:45:54.504808+0000","last_became_active":"2024-09-13T16:43:54.559586+0000","last_became_peered":"2024-09-13T16:43:54.559586+0000","last_unstale":"2024-09-13T16:45:54.504808+0000","last_undegraded":"2024-09-13T16:45:54.504808+0000","last_fullsized":"2024-09-13T16:45:54.504808+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"22'147","reported_seq":161,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.520197+0000","last_change":"2024-09-13T16:43:54.996099+0000","last_active":"2024-09-13T16:47:59.520197+0000","last_peered":"2024-09-13T16:47:59.520197+0000","last_clean":"2024-09-13T16:47:59.520197+0000","last_became_active":"2024-09-13T16:43:54.995802+0000","last_became_peered":"2024-09-13T16:43:54.995802+0000","last_unstale":"2024-09-13T16:47:59.520197+0000","last_undegraded":"2024-09-13T16:47:59.520197+0000","last_fullsized":"2024-09-13T16:47:59.520197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":28,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:59.505735+0000","last_change":"2024-09-13T16:43:54.562699+0000","last_active":"2024-09-13T16:45:59.505735+0000","last_peered":"2024-09-13T16:45:59.505735+0000","last_clean":"2024-09-13T16:45:59.505735+0000","last_became_active":"2024-09-13T16:43:54.560552+0000","last_became_peered":"2024-09-13T16:43:54.560552+0000","last_unstale":"2024-09-13T16:45:59.505735+0000","last_undegraded":"2024-09-13T16:45:59.505735+0000","last_fullsized":"2024-09-13T16:45:59.505735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:11.032382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10755,"ondisk_log_size":10755,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":371443,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":402,"num_write_kb":563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":742886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":367,"ondisk_log_size":367,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574904,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39628,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332212,"statfs":{"total":96636764160,"available":96596185088,"internally_reserved":0,"allocated":331776,"data_stored":675500,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":40237514},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.59199999999999997,"15min":0.57299999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":5.0620000000000003,"5min":5.0620000000000003,"15min":5.0620000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.68500000000000005,"15min":0.60099999999999998},"min":{"1min":0.372,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":5.1859999999999999,"5min":5.1859999999999999,"15min":5.1859999999999999},"last":0.46800000000000003}]},{"osd":1,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.36799999999999999,"5min":0.314,"15min":0.314},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.61699999999999999,"15min":0.58999999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":3.6070000000000002,"5min":3.6070000000000002,"15min":3.6070000000000002},"last":0.60499999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":2.4929999999999999,"5min":2.4929999999999999,"15min":2.4929999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.63400000000000001,"15min":0.61599999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":0.51200000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.56899999999999995,"15min":0.58199999999999996},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.8490000000000002,"5min":3.8490000000000002,"15min":3.8490000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64800000000000002,"15min":0.60499999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.3160000000000001,"5min":4.0129999999999999,"15min":4.0129999999999999},"last":0.57399999999999995}]},{"osd":4,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.434,"15min":0.442},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.78000000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.37},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45000000000000001,"15min":0.49199999999999999},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67300000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.45100000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46300000000000002,"15min":0.48699999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.499,"15min":0.52200000000000002},"min":{"1min":0.32600000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.753,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.40699999999999997}]},{"osd":6,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.53300000000000003,"15min":0.56299999999999994},"min":{"1min":0.36399999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81100000000000005,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.44},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.504,"15min":0.54000000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72799999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.497}]}]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35812,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336028,"statfs":{"total":96636764160,"available":96600092672,"internally_reserved":0,"allocated":618496,"data_stored":448530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36043209},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":0.69499999999999995,"15min":0.625},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":8.468,"5min":8.468,"15min":8.468},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.67300000000000004,"15min":0.625},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":7.5739999999999998,"5min":7.5739999999999998,"15min":7.5739999999999998},"last":0.76900000000000002}]},{"osd":1,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.29299999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60799999999999998,"15min":0.63900000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.67400000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.028,"5min":2.431,"15min":2.431},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.466,"5min":0.78500000000000003,"15min":0.64100000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.9139999999999997,"5min":6.9139999999999997,"15min":6.9139999999999997},"last":0.84999999999999998}]},{"osd":3,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":0.67200000000000004,"15min":0.60999999999999999},"min":{"1min":0.373,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.2400000000000002,"5min":6.2400000000000002,"15min":6.2400000000000002},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.70899999999999996,"15min":0.66200000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.4500000000000002,"5min":3.4500000000000002,"15min":3.4500000000000002},"last":0.86699999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.41299999999999998,"15min":0.45100000000000001},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.54300000000000004,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.439,"15min":0.48299999999999998},"min":{"1min":0.23699999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.441}]},{"osd":5,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.48799999999999999,"15min":0.52800000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.51100000000000001,"15min":0.56499999999999995},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.73599999999999999,"5min":0.752,"15min":0.752},"last":0.49099999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.48199999999999998,"15min":0.52000000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.71299999999999997,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.54000000000000004,"15min":0.58099999999999996},"min":{"1min":0.34000000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.53600000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574904,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39572,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332268,"statfs":{"total":96636764160,"available":96596242432,"internally_reserved":0,"allocated":274432,"data_stored":86302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.58299999999999996,"15min":0.56299999999999994},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.411,"5min":3.411,"15min":3.411},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.117,"5min":3.117,"15min":3.117},"last":0.57099999999999995}]},{"osd":1,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55500000000000005,"15min":0.56100000000000005},"min":{"1min":0.29899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.64300000000000002,"15min":0.58899999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.843,"5min":4.3860000000000001,"15min":4.3860000000000001},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.67300000000000004,"15min":0.64300000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":4.0819999999999999,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.64700000000000002,"15min":0.61399999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":4.3170000000000002,"5min":4.3170000000000002,"15min":4.3170000000000002},"last":0.64900000000000002}]},{"osd":3,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.90700000000000003,"15min":0.68700000000000006},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.4690000000000003,"5min":28.707000000000001,"15min":28.707000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.68700000000000006,"15min":0.625},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.2599999999999998,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":0.58499999999999996}]},{"osd":4,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.41899999999999998,"15min":0.45100000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.45500000000000002,"15min":0.48099999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.70399999999999996,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.39000000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.441,"15min":0.45700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.67000000000000004,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.46500000000000002,"15min":0.48499999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.40600000000000003}]},{"osd":7,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.49099999999999999,"15min":0.52000000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65600000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.501,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.42499999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574905,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39552,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332288,"statfs":{"total":96636764160,"available":96596262912,"internally_reserved":0,"allocated":253952,"data_stored":84621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":3.7160000000000002,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.61099999999999999,"15min":0.59399999999999997},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.448,"5min":2.9980000000000002,"15min":2.9980000000000002},"last":0.61199999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.69599999999999995,"15min":0.63},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.3700000000000001,"5min":4.3700000000000001,"15min":4.3700000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.63500000000000001,"15min":0.61499999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.9020000000000001,"5min":3.9020000000000001,"15min":3.9020000000000001},"last":0.56399999999999995}]},{"osd":2,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":2.085,"5min":2.613,"15min":2.613},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.194,"5min":0.72499999999999998,"15min":0.65800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":7.9809999999999999,"5min":7.9809999999999999,"15min":7.9809999999999999},"last":0.54000000000000004}]},{"osd":3,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.65800000000000003,"15min":0.63400000000000001},"min":{"1min":0.36199999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.68600000000000005,"15min":0.68899999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.8130000000000002,"5min":2.8130000000000002,"15min":2.8130000000000002},"last":0.65100000000000002}]},{"osd":5,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.42799999999999999,"15min":0.46100000000000002},"min":{"1min":0.25600000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.373},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.438,"15min":0.47299999999999998},"min":{"1min":0.254,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61399999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.41099999999999998}]},{"osd":6,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.442,"15min":0.47999999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.628,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.48599999999999999,"15min":0.53400000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.63600000000000001,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.434}]},{"osd":7,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.48099999999999998,"15min":0.50900000000000001},"min":{"1min":0.308,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.499},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.54400000000000004,"15min":0.60399999999999998},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70499999999999996,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.54700000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574904,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39916,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331924,"statfs":{"total":96636764160,"available":96595890176,"internally_reserved":0,"allocated":626688,"data_stored":448810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40237512},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.57899999999999996,"15min":0.49199999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.8449999999999998,"5min":4.8449999999999998,"15min":4.8449999999999998},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.5529999999999999,"5min":0.77800000000000002,"15min":0.59799999999999998},"min":{"1min":0.41099999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":7.7489999999999997,"5min":7.7489999999999997,"15min":7.7489999999999997},"last":0.42599999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":2.0529999999999999,"5min":0.84299999999999997,"15min":0.60499999999999998},"min":{"1min":0.317,"5min":0.252,"15min":0.252},"max":{"1min":28.908999999999999,"5min":28.908999999999999,"15min":28.908999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":0.73299999999999998,"15min":0.64600000000000002},"min":{"1min":0.40600000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.1520000000000001,"5min":3.1699999999999999,"15min":3.1699999999999999},"last":0.45300000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":0.80300000000000005,"15min":0.63500000000000001},"min":{"1min":0.41799999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":4.3630000000000004,"5min":4.3630000000000004,"15min":4.3630000000000004},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.76900000000000002,"15min":0.69299999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.1859999999999999,"5min":3.2610000000000001,"15min":3.2610000000000001},"last":0.60499999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":0.94299999999999995,"15min":0.82699999999999996},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":4.1029999999999998,"5min":4.1029999999999998,"15min":4.1029999999999998},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.79300000000000004,"15min":0.753},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.903,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":0.754}]},{"osd":5,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":0.80700000000000005,"15min":0.754},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":2.012,"5min":2.5790000000000002,"15min":2.5790000000000002},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.1160000000000001,"5min":0.89100000000000001,"15min":0.84699999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.0739999999999998,"5min":3.1970000000000001,"15min":3.1970000000000001},"last":0.71699999999999997}]},{"osd":6,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":0.96699999999999997,"15min":0.84199999999999997},"min":{"1min":0.61099999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":4.1449999999999996,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.0760000000000001,"5min":0.86499999999999999,"15min":0.82799999999999996},"min":{"1min":0.53000000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.9319999999999999,"5min":2.7360000000000002,"15min":2.7360000000000002},"last":0.79800000000000004}]},{"osd":7,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":0.88500000000000001,"15min":0.82399999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":3.117,"5min":3.2320000000000002,"15min":3.2320000000000002},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.613,"5min":1.006,"15min":0.874},"min":{"1min":0.56599999999999995,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":4.181,"5min":4.181,"15min":4.181},"last":0.86399999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35464,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336376,"statfs":{"total":96636764160,"available":96600449024,"internally_reserved":0,"allocated":262144,"data_stored":84385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.988,"5min":0.996,"15min":0.65800000000000003},"min":{"1min":0.26600000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":31.440999999999999,"5min":31.440999999999999,"15min":31.440999999999999},"last":0.437},{"interface":"front","average":{"1min":1.294,"5min":0.69099999999999995,"15min":0.57699999999999996},"min":{"1min":0.34899999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.7720000000000002,"5min":4.7720000000000002,"15min":4.7720000000000002},"last":0.47199999999999998}]},{"osd":1,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.2949999999999999,"5min":0.89800000000000002,"15min":0.65600000000000003},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.497},{"interface":"front","average":{"1min":1.121,"5min":0.755,"15min":0.61299999999999999},"min":{"1min":0.379,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.0230000000000001,"5min":6.5659999999999998,"15min":6.5659999999999998},"last":0.59099999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":0.71499999999999997,"15min":0.59299999999999997},"min":{"1min":0.371,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":6.5469999999999997,"5min":6.5469999999999997,"15min":6.5469999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.7869999999999999,"5min":0.84499999999999997,"15min":0.68300000000000005},"min":{"1min":0.45500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":8.8659999999999997,"5min":8.8659999999999997,"15min":8.8659999999999997},"last":0.57899999999999996}]},{"osd":4,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":0.81499999999999995,"15min":0.752},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":3.4460000000000002,"5min":3.4460000000000002,"15min":3.4460000000000002},"last":0.63},{"interface":"front","average":{"1min":1.163,"5min":0.80600000000000005,"15min":0.76300000000000001},"min":{"1min":0.46200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":6.4379999999999997,"5min":6.4379999999999997,"15min":6.4379999999999997},"last":0.76800000000000002}]},{"osd":5,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.929,"5min":0.97599999999999998,"15min":0.81699999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":9.1039999999999992,"5min":9.1039999999999992,"15min":9.1039999999999992},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.1899999999999999,"5min":0.84399999999999997,"15min":0.77400000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":3.633,"5min":3.633,"15min":3.633},"last":0.75}]},{"osd":6,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":0.83199999999999996,"15min":0.79400000000000004},"min":{"1min":0.54400000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.95,"5min":0.999,"15min":0.85899999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":9.141,"5min":9.141,"15min":9.141},"last":0.79200000000000004}]},{"osd":7,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":0.88400000000000001,"15min":0.84299999999999997},"min":{"1min":0.50700000000000001,"5min":0.505,"15min":0.505},"max":{"1min":3.5990000000000002,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.2450000000000001,"5min":0.86699999999999999,"15min":0.83499999999999996},"min":{"1min":0.57599999999999996,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":6.4800000000000004,"5min":6.4800000000000004,"15min":6.4800000000000004},"last":0.84199999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35460,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336380,"statfs":{"total":96636764160,"available":96600453120,"internally_reserved":0,"allocated":258048,"data_stored":84564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5630000000000002,"5min":1.0009999999999999,"15min":0.65200000000000002},"min":{"1min":0.29399999999999998,"5min":0.249,"15min":0.249},"max":{"1min":22.721,"5min":22.721,"15min":22.721},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.226,"5min":0.86699999999999999,"15min":0.64100000000000001},"min":{"1min":0.32200000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.2430000000000003,"5min":19.494,"15min":19.494},"last":0.69699999999999995}]},{"osd":2,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3570000000000002,"5min":0.98999999999999999,"15min":0.67200000000000004},"min":{"1min":0.34799999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":15.491,"5min":15.491,"15min":15.491},"last":0.57999999999999996},{"interface":"front","average":{"1min":1.5549999999999999,"5min":0.93300000000000005,"15min":0.70399999999999996},"min":{"1min":0.39600000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.6829999999999998,"5min":17.071000000000002,"15min":17.071000000000002},"last":0.50800000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.631,"5min":0.98599999999999999,"15min":0.73499999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":6.883,"5min":18.838000000000001,"15min":18.838000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.339,"5min":0.92200000000000004,"15min":0.69699999999999995},"min":{"1min":0.40899999999999997,"5min":0.31,"15min":0.31},"max":{"1min":7.0339999999999998,"5min":20.199000000000002,"15min":20.199000000000002},"last":0.63}]},{"osd":4,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.778,"5min":1.2390000000000001,"15min":0.91300000000000003},"min":{"1min":0.39800000000000002,"5min":0.373,"15min":0.373},"max":{"1min":15.654999999999999,"5min":15.654999999999999,"15min":15.654999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.091,"15min":0.89500000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":7.5659999999999998,"5min":18.742000000000001,"15min":18.742000000000001},"last":0.82099999999999995}]},{"osd":5,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":0.97699999999999998,"15min":0.89300000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":7.3230000000000004,"5min":7.3230000000000004,"15min":7.3230000000000004},"last":0.84599999999999997},{"interface":"front","average":{"1min":2.819,"5min":1.296,"15min":0.98299999999999998},"min":{"1min":0.51400000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":15.686,"5min":15.686,"15min":15.686},"last":0.87}]},{"osd":6,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":0.96799999999999997,"15min":0.86599999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":7.3700000000000001,"5min":7.3700000000000001,"15min":7.3700000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":2.851,"5min":1.3180000000000001,"15min":1.002},"min":{"1min":0.55500000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":15.718,"5min":15.718,"15min":15.718},"last":0.83799999999999997}]},{"osd":7,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":1.119,"15min":0.89500000000000002},"min":{"1min":0.495,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":8.1539999999999999,"5min":18.798999999999999,"15min":18.798999999999999},"last":0.88},{"interface":"front","average":{"1min":1.4339999999999999,"5min":0.995,"15min":0.90200000000000002},"min":{"1min":0.53000000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":7.407,"5min":7.4950000000000001,"15min":7.4950000000000001},"last":0.76000000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574904,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39616,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332224,"statfs":{"total":96636764160,"available":96596197376,"internally_reserved":0,"allocated":319488,"data_stored":673303,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.65800000000000003,"15min":0.53000000000000003},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.9239999999999999,"5min":7.2539999999999996,"15min":7.2539999999999996},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.2589999999999999,"5min":0.71999999999999997,"15min":0.55700000000000005},"min":{"1min":0.32400000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":6.0949999999999998,"5min":6.0949999999999998,"15min":6.0949999999999998},"last":0.68899999999999995}]},{"osd":2,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.78300000000000003,"15min":0.627},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.1800000000000002,"5min":7.1920000000000002,"15min":7.1920000000000002},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.151,"5min":0.74299999999999999,"15min":0.59899999999999998},"min":{"1min":0.36399999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":6.5,"5min":6.5,"15min":6.5},"last":0.51200000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":0.66500000000000004,"15min":0.54800000000000004},"min":{"1min":0.38800000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":4.4459999999999997,"5min":4.4459999999999997,"15min":4.4459999999999997},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.85999999999999999,"15min":0.69099999999999995},"min":{"1min":0.38700000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0409999999999999,"5min":10.522,"15min":10.522},"last":0.59799999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":0.89300000000000002,"15min":0.80100000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":6.4560000000000004,"5min":6.4560000000000004,"15min":6.4560000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.82799999999999996,"15min":0.77900000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":3.79,"5min":3.79,"15min":3.79},"last":0.76000000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":0.85399999999999998,"15min":0.78800000000000003},"min":{"1min":0.51300000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.8340000000000001,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":1.002,"5min":0.96599999999999997,"15min":0.85399999999999998},"min":{"1min":0.48599999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":4.5620000000000003,"5min":7.8339999999999996,"15min":7.8339999999999996},"last":0.70799999999999996}]},{"osd":6,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.98199999999999998,"15min":0.86499999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.7270000000000003,"5min":7.5510000000000002,"15min":7.5510000000000002},"last":0.754},{"interface":"front","average":{"1min":1.345,"5min":0.95699999999999996,"15min":0.81499999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":7.444,"5min":7.444,"15min":7.444},"last":0.78000000000000003}]},{"osd":7,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":0.95399999999999996,"15min":0.81599999999999995},"min":{"1min":0.51900000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":7.5350000000000001,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.1639999999999999,"5min":0.874,"15min":0.79700000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.1879999999999997,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":365730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":389120,"data_stored":365450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:48:23.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:48:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.379+0000 7fb7f4e00640 1 Processor -- start 2024-09-13T16:48:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.379+0000 7fb7f4e00640 1 -- start start 2024-09-13T16:48:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.380+0000 7fb7f4e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7f014f410 0x7fb7f00c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.380+0000 7fb7f4e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb7f0070540 0x7fb7f015f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.380+0000 7fb7f4e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f0155ef0 con 0x7fb7f01559f0 2024-09-13T16:48:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.380+0000 7fb7f4e00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f00c2230 con 0x7fb7f0070540 2024-09-13T16:48:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.380+0000 7fb7f4e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f00c63e0 con 0x7fb7f014f410 2024-09-13T16:48:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.381+0000 7fb7ed400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb7f01559f0 0x7fb7f006d8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52730/0 (socket says 172.21.15.28:52730) 2024-09-13T16:48:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.381+0000 7fb7ed400640 1 -- 172.21.15.28:0/3390398245 learned_addr learned my addr 172.21.15.28:0/3390398245 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.381+0000 7fb7ee800640 1 --2- 172.21.15.28:0/3390398245 >> v2:172.21.15.191:3300/0 conn(0x7fb7f0070540 0x7fb7f015f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.381+0000 7fb7ede00640 1 --2- 172.21.15.28:0/3390398245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7f014f410 0x7fb7f00c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.381+0000 7fb7eca00640 1 -- 172.21.15.28:0/3390398245 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275924557 0 0) 0x7fb7f0155ef0 con 0x7fb7f01559f0 2024-09-13T16:48:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.381+0000 7fb7eca00640 1 -- 172.21.15.28:0/3390398245 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d4005730 con 0x7fb7f01559f0 2024-09-13T16:48:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.381+0000 7fb7ede00640 1 -- 172.21.15.28:0/3390398245 >> v2:172.21.15.191:3300/0 conn(0x7fb7f0070540 msgr2=0x7fb7f015f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.381+0000 7fb7ede00640 1 --2- 172.21.15.28:0/3390398245 >> v2:172.21.15.191:3300/0 conn(0x7fb7f0070540 0x7fb7f015f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.381+0000 7fb7ede00640 1 -- 172.21.15.28:0/3390398245 >> v1:172.21.15.28:6789/0 conn(0x7fb7f01559f0 legacy=0x7fb7f006d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.381+0000 7fb7ede00640 1 -- 172.21.15.28:0/3390398245 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f00c4e00 con 0x7fb7f014f410 2024-09-13T16:48:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.381+0000 7fb7ee800640 1 --2- 172.21.15.28:0/3390398245 >> v2:172.21.15.191:3300/0 conn(0x7fb7f0070540 0x7fb7f015f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.382+0000 7fb7ede00640 1 --2- 172.21.15.28:0/3390398245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7f014f410 0x7fb7f00c5ea0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb7e00097a0 tx=0x7fb7e002a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2817f4f5d98465e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:23.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.382+0000 7fb7eca00640 1 -- 172.21.15.28:0/3390398245 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0032440 con 0x7fb7f014f410 2024-09-13T16:48:23.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.383+0000 7fb7eca00640 1 -- 172.21.15.28:0/3390398245 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e00325a0 con 0x7fb7f014f410 2024-09-13T16:48:23.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.383+0000 7fb7eca00640 1 -- 172.21.15.28:0/3390398245 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0032930 con 0x7fb7f014f410 2024-09-13T16:48:23.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.383+0000 7fb7f4e00640 1 -- 172.21.15.28:0/3390398245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7f014f410 msgr2=0x7fb7f00c5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:23.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.383+0000 7fb7f4e00640 1 --2- 172.21.15.28:0/3390398245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7f014f410 0x7fb7f00c5ea0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb7e00097a0 tx=0x7fb7e002a230 comp rx=0 tx=0).stop 2024-09-13T16:48:23.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.383+0000 7fb7f4e00640 1 -- 172.21.15.28:0/3390398245 shutdown_connections 2024-09-13T16:48:23.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.383+0000 7fb7f4e00640 1 --2- 172.21.15.28:0/3390398245 >> v2:172.21.15.191:3300/0 conn(0x7fb7f0070540 0x7fb7f015f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.383+0000 7fb7f4e00640 1 --2- 172.21.15.28:0/3390398245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7f014f410 0x7fb7f00c5ea0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.383+0000 7fb7f4e00640 1 -- 172.21.15.28:0/3390398245 >> 172.21.15.28:0/3390398245 conn(0x7fb7f0066040 msgr2=0x7fb7f0069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:23.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.383+0000 7fb7f4e00640 1 -- 172.21.15.28:0/3390398245 shutdown_connections 2024-09-13T16:48:23.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.383+0000 7fb7f4e00640 1 -- 172.21.15.28:0/3390398245 wait complete. 2024-09-13T16:48:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.385+0000 7fb7f4e00640 1 Processor -- start 2024-09-13T16:48:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.385+0000 7fb7f4e00640 1 -- start start 2024-09-13T16:48:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.386+0000 7fb7f4e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7f0070540 0x7fb7f01f0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.386+0000 7fb7f4e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7f014f410 0x7fb7f00bc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.386+0000 7fb7f4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f01559f0 0x7fb7f00bc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.386+0000 7fb7f4e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f0146e40 con 0x7fb7f01559f0 2024-09-13T16:48:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.386+0000 7fb7f4e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f00c0ef0 con 0x7fb7f014f410 2024-09-13T16:48:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.386+0000 7fb7f4e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f00c1140 con 0x7fb7f0070540 2024-09-13T16:48:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.386+0000 7fb7ee800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f01559f0 0x7fb7f00bc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.386+0000 7fb7ede00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7f0070540 0x7fb7f01f0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.386+0000 7fb7ed400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7f014f410 0x7fb7f00bc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.386+0000 7fb7ee800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f01559f0 0x7fb7f00bc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39150/0 (socket says 172.21.15.28:39150) 2024-09-13T16:48:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.386+0000 7fb7ee800640 1 -- 172.21.15.28:0/1255521847 learned_addr learned my addr 172.21.15.28:0/1255521847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.387+0000 7fb7ede00640 1 -- 172.21.15.28:0/1255521847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7f014f410 msgr2=0x7fb7f00bc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.387+0000 7fb7ede00640 1 --2- 172.21.15.28:0/1255521847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7f014f410 0x7fb7f00bc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.387+0000 7fb7ede00640 1 -- 172.21.15.28:0/1255521847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f01559f0 msgr2=0x7fb7f00bc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.387+0000 7fb7ede00640 1 --2- 172.21.15.28:0/1255521847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f01559f0 0x7fb7f00bc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.387+0000 7fb7ede00640 1 -- 172.21.15.28:0/1255521847 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e800ab80 con 0x7fb7f0070540 2024-09-13T16:48:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.387+0000 7fb7ee800640 1 --2- 172.21.15.28:0/1255521847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f01559f0 0x7fb7f00bc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.387+0000 7fb7ede00640 1 --2- 172.21.15.28:0/1255521847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7f0070540 0x7fb7f01f0b90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0031000 tx=0x7fb7e00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.387+0000 7fb7e6000640 1 -- 172.21.15.28:0/1255521847 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0030b80 con 0x7fb7f0070540 2024-09-13T16:48:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.387+0000 7fb7e6000640 1 -- 172.21.15.28:0/1255521847 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e0030d20 con 0x7fb7f0070540 2024-09-13T16:48:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.387+0000 7fb7f4e00640 1 -- 172.21.15.28:0/1255521847 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e0009400 con 0x7fb7f0070540 2024-09-13T16:48:23.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.387+0000 7fb7e6000640 1 -- 172.21.15.28:0/1255521847 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e003b640 con 0x7fb7f0070540 2024-09-13T16:48:23.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.388+0000 7fb7f4e00640 1 -- 172.21.15.28:0/1255521847 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f00bf540 con 0x7fb7f0070540 2024-09-13T16:48:23.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.389+0000 7fb7f4e00640 1 -- 172.21.15.28:0/1255521847 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f00c6cc0 con 0x7fb7f0070540 2024-09-13T16:48:23.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.390+0000 7fb7e6000640 1 -- 172.21.15.28:0/1255521847 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb7e0039040 con 0x7fb7f0070540 2024-09-13T16:48:23.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.390+0000 7fb7e6000640 1 --2- 172.21.15.28:0/1255521847 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb7d006ac00 0x7fb7d006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:23.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.391+0000 7fb7e6000640 1 -- 172.21.15.28:0/1255521847 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb7e00a99d0 con 0x7fb7f0070540 2024-09-13T16:48:23.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.391+0000 7fb7ed400640 1 --2- 172.21.15.28:0/1255521847 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb7d006ac00 0x7fb7d006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:23.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.391+0000 7fb7ed400640 1 --2- 172.21.15.28:0/1255521847 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb7d006ac00 0x7fb7d006d0c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0152040 tx=0x7fb7dc007d30 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:23.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.395+0000 7fb7e6000640 1 -- 172.21.15.28:0/1255521847 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb7e0074bd0 con 0x7fb7f0070540 2024-09-13T16:48:23.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.570+0000 7fb7f4e00640 1 -- 172.21.15.28:0/1255521847 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb7f00c56a0 con 0x7fb7d006ac00 2024-09-13T16:48:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.578+0000 7fb7e6000640 1 -- 172.21.15.28:0/1255521847 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289778 (secure 0 0 0) 0x7fb7dc0014d0 con 0x7fb7d006ac00 2024-09-13T16:48:23.583 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:48:23.584 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T16:48:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.581+0000 7fb7f4e00640 1 -- 172.21.15.28:0/1255521847 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb7d006ac00 msgr2=0x7fb7d006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.581+0000 7fb7f4e00640 1 --2- 172.21.15.28:0/1255521847 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb7d006ac00 0x7fb7d006d0c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0152040 tx=0x7fb7dc007d30 comp rx=0 tx=0).stop 2024-09-13T16:48:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.581+0000 7fb7f4e00640 1 -- 172.21.15.28:0/1255521847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7f0070540 msgr2=0x7fb7f01f0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.581+0000 7fb7f4e00640 1 --2- 172.21.15.28:0/1255521847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7f0070540 0x7fb7f01f0b90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0031000 tx=0x7fb7e00046c0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.581+0000 7fb7f4e00640 1 -- 172.21.15.28:0/1255521847 shutdown_connections 2024-09-13T16:48:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.581+0000 7fb7f4e00640 1 --2- 172.21.15.28:0/1255521847 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb7d006ac00 0x7fb7d006d0c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.581+0000 7fb7f4e00640 1 --2- 172.21.15.28:0/1255521847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f01559f0 0x7fb7f00bc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.581+0000 7fb7f4e00640 1 --2- 172.21.15.28:0/1255521847 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb7f014f410 0x7fb7f00bc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.581+0000 7fb7f4e00640 1 --2- 172.21.15.28:0/1255521847 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb7f0070540 0x7fb7f01f0b90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.581+0000 7fb7f4e00640 1 -- 172.21.15.28:0/1255521847 >> 172.21.15.28:0/1255521847 conn(0x7fb7f0066040 msgr2=0x7fb7f0071e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.582+0000 7fb7f4e00640 1 -- 172.21.15.28:0/1255521847 shutdown_connections 2024-09-13T16:48:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.582+0000 7fb7f4e00640 1 -- 172.21.15.28:0/1255521847 wait complete. 2024-09-13T16:48:23.598 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-09-13T16:48:21.652636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":961811,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":121,"num_read_kb":105,"num_write":9777,"num_write_kb":25168803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11239,"ondisk_log_size":11239,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":305020,"kb_used_data":2876,"kb_used_omap":12,"kb_used_meta":302067,"kb_avail":754669700,"statfs":{"total":773094113280,"available":772781772800,"internally_reserved":0,"allocated":2945024,"data_stored":2586015,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12743,"internal_metadata":309317177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":700,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003045"},"pg_stats":[{"pgid":"3.3f","version":"22'179","reported_seq":193,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.519062+0000","last_change":"2024-09-13T16:43:54.987241+0000","last_active":"2024-09-13T16:47:59.519062+0000","last_peered":"2024-09-13T16:47:59.519062+0000","last_clean":"2024-09-13T16:47:59.519062+0000","last_became_active":"2024-09-13T16:43:54.986988+0000","last_became_peered":"2024-09-13T16:43:54.986988+0000","last_unstale":"2024-09-13T16:47:59.519062+0000","last_undegraded":"2024-09-13T16:47:59.519062+0000","last_fullsized":"2024-09-13T16:47:59.519062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"22'134","reported_seq":148,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.454727+0000","last_change":"2024-09-13T16:43:55.001141+0000","last_active":"2024-09-13T16:47:59.454727+0000","last_peered":"2024-09-13T16:47:59.454727+0000","last_clean":"2024-09-13T16:47:59.454727+0000","last_became_active":"2024-09-13T16:43:55.000757+0000","last_became_peered":"2024-09-13T16:43:55.000757+0000","last_unstale":"2024-09-13T16:47:59.454727+0000","last_undegraded":"2024-09-13T16:47:59.454727+0000","last_fullsized":"2024-09-13T16:47:59.454727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"22'205","reported_seq":219,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.510237+0000","last_change":"2024-09-13T16:43:54.986925+0000","last_active":"2024-09-13T16:47:59.510237+0000","last_peered":"2024-09-13T16:47:59.510237+0000","last_clean":"2024-09-13T16:47:59.510237+0000","last_became_active":"2024-09-13T16:43:54.986198+0000","last_became_peered":"2024-09-13T16:43:54.986198+0000","last_unstale":"2024-09-13T16:47:59.510237+0000","last_undegraded":"2024-09-13T16:47:59.510237+0000","last_fullsized":"2024-09-13T16:47:59.510237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:03.127550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"22'125","reported_seq":139,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.433692+0000","last_change":"2024-09-13T16:43:54.992249+0000","last_active":"2024-09-13T16:47:59.433692+0000","last_peered":"2024-09-13T16:47:59.433692+0000","last_clean":"2024-09-13T16:47:59.433692+0000","last_became_active":"2024-09-13T16:43:54.991804+0000","last_became_peered":"2024-09-13T16:43:54.991804+0000","last_unstale":"2024-09-13T16:47:59.433692+0000","last_undegraded":"2024-09-13T16:47:59.433692+0000","last_fullsized":"2024-09-13T16:47:59.433692+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":270336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"22'190","reported_seq":203,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.430620+0000","last_change":"2024-09-13T16:43:54.987564+0000","last_active":"2024-09-13T16:47:59.430620+0000","last_peered":"2024-09-13T16:47:59.430620+0000","last_clean":"2024-09-13T16:47:59.430620+0000","last_became_active":"2024-09-13T16:43:54.987328+0000","last_became_peered":"2024-09-13T16:43:54.987328+0000","last_unstale":"2024-09-13T16:47:59.430620+0000","last_undegraded":"2024-09-13T16:47:59.430620+0000","last_fullsized":"2024-09-13T16:47:59.430620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:36.006730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"22'150","reported_seq":163,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.494433+0000","last_change":"2024-09-13T16:43:55.000019+0000","last_active":"2024-09-13T16:47:59.494433+0000","last_peered":"2024-09-13T16:47:59.494433+0000","last_clean":"2024-09-13T16:47:59.494433+0000","last_became_active":"2024-09-13T16:43:54.999752+0000","last_became_peered":"2024-09-13T16:43:54.999752+0000","last_unstale":"2024-09-13T16:47:59.494433+0000","last_undegraded":"2024-09-13T16:47:59.494433+0000","last_fullsized":"2024-09-13T16:47:59.494433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"22'170","reported_seq":183,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.493499+0000","last_change":"2024-09-13T16:43:54.999533+0000","last_active":"2024-09-13T16:47:59.493499+0000","last_peered":"2024-09-13T16:47:59.493499+0000","last_clean":"2024-09-13T16:47:59.493499+0000","last_became_active":"2024-09-13T16:43:54.999350+0000","last_became_peered":"2024-09-13T16:43:54.999350+0000","last_unstale":"2024-09-13T16:47:59.493499+0000","last_undegraded":"2024-09-13T16:47:59.493499+0000","last_fullsized":"2024-09-13T16:47:59.493499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:29.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"22'156","reported_seq":169,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.511303+0000","last_change":"2024-09-13T16:43:54.998526+0000","last_active":"2024-09-13T16:47:59.511303+0000","last_peered":"2024-09-13T16:47:59.511303+0000","last_clean":"2024-09-13T16:47:59.511303+0000","last_became_active":"2024-09-13T16:43:54.996309+0000","last_became_peered":"2024-09-13T16:43:54.996309+0000","last_unstale":"2024-09-13T16:47:59.511303+0000","last_undegraded":"2024-09-13T16:47:59.511303+0000","last_fullsized":"2024-09-13T16:47:59.511303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"22'144","reported_seq":157,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.520032+0000","last_change":"2024-09-13T16:43:54.993786+0000","last_active":"2024-09-13T16:47:59.520032+0000","last_peered":"2024-09-13T16:47:59.520032+0000","last_clean":"2024-09-13T16:47:59.520032+0000","last_became_active":"2024-09-13T16:43:54.993475+0000","last_became_peered":"2024-09-13T16:43:54.993475+0000","last_unstale":"2024-09-13T16:47:59.520032+0000","last_undegraded":"2024-09-13T16:47:59.520032+0000","last_fullsized":"2024-09-13T16:47:59.520032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"22'174","reported_seq":187,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.513207+0000","last_change":"2024-09-13T16:43:54.986272+0000","last_active":"2024-09-13T16:47:59.513207+0000","last_peered":"2024-09-13T16:47:59.513207+0000","last_clean":"2024-09-13T16:47:59.513207+0000","last_became_active":"2024-09-13T16:43:54.985979+0000","last_became_peered":"2024-09-13T16:43:54.985979+0000","last_unstale":"2024-09-13T16:47:59.513207+0000","last_undegraded":"2024-09-13T16:47:59.513207+0000","last_fullsized":"2024-09-13T16:47:59.513207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"22'168","reported_seq":181,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.448009+0000","last_change":"2024-09-13T16:43:54.986207+0000","last_active":"2024-09-13T16:47:59.448009+0000","last_peered":"2024-09-13T16:47:59.448009+0000","last_clean":"2024-09-13T16:47:59.448009+0000","last_became_active":"2024-09-13T16:43:54.985948+0000","last_became_peered":"2024-09-13T16:43:54.985948+0000","last_unstale":"2024-09-13T16:47:59.448009+0000","last_undegraded":"2024-09-13T16:47:59.448009+0000","last_fullsized":"2024-09-13T16:47:59.448009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"22'176","reported_seq":189,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.438078+0000","last_change":"2024-09-13T16:43:54.986636+0000","last_active":"2024-09-13T16:47:59.438078+0000","last_peered":"2024-09-13T16:47:59.438078+0000","last_clean":"2024-09-13T16:47:59.438078+0000","last_became_active":"2024-09-13T16:43:54.986161+0000","last_became_peered":"2024-09-13T16:43:54.986161+0000","last_unstale":"2024-09-13T16:47:59.438078+0000","last_undegraded":"2024-09-13T16:47:59.438078+0000","last_fullsized":"2024-09-13T16:47:59.438078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:09.340330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"22'160","reported_seq":173,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.511135+0000","last_change":"2024-09-13T16:43:54.997783+0000","last_active":"2024-09-13T16:47:59.511135+0000","last_peered":"2024-09-13T16:47:59.511135+0000","last_clean":"2024-09-13T16:47:59.511135+0000","last_became_active":"2024-09-13T16:43:54.997162+0000","last_became_peered":"2024-09-13T16:43:54.997162+0000","last_unstale":"2024-09-13T16:47:59.511135+0000","last_undegraded":"2024-09-13T16:47:59.511135+0000","last_fullsized":"2024-09-13T16:47:59.511135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"22'173","reported_seq":186,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.497364+0000","last_change":"2024-09-13T16:43:54.995262+0000","last_active":"2024-09-13T16:47:59.497364+0000","last_peered":"2024-09-13T16:47:59.497364+0000","last_clean":"2024-09-13T16:47:59.497364+0000","last_became_active":"2024-09-13T16:43:54.994891+0000","last_became_peered":"2024-09-13T16:43:54.994891+0000","last_unstale":"2024-09-13T16:47:59.497364+0000","last_undegraded":"2024-09-13T16:47:59.497364+0000","last_fullsized":"2024-09-13T16:47:59.497364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"22'185","reported_seq":198,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.492832+0000","last_change":"2024-09-13T16:43:54.981534+0000","last_active":"2024-09-13T16:47:59.492832+0000","last_peered":"2024-09-13T16:47:59.492832+0000","last_clean":"2024-09-13T16:47:59.492832+0000","last_became_active":"2024-09-13T16:43:54.981191+0000","last_became_peered":"2024-09-13T16:43:54.981191+0000","last_unstale":"2024-09-13T16:47:59.492832+0000","last_undegraded":"2024-09-13T16:47:59.492832+0000","last_fullsized":"2024-09-13T16:47:59.492832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"22'151","reported_seq":164,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.506334+0000","last_change":"2024-09-13T16:43:54.994157+0000","last_active":"2024-09-13T16:47:59.506334+0000","last_peered":"2024-09-13T16:47:59.506334+0000","last_clean":"2024-09-13T16:47:59.506334+0000","last_became_active":"2024-09-13T16:43:54.993915+0000","last_became_peered":"2024-09-13T16:43:54.993915+0000","last_unstale":"2024-09-13T16:47:59.506334+0000","last_undegraded":"2024-09-13T16:47:59.506334+0000","last_fullsized":"2024-09-13T16:47:59.506334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"22'167","reported_seq":180,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.509298+0000","last_change":"2024-09-13T16:43:54.987328+0000","last_active":"2024-09-13T16:47:59.509298+0000","last_peered":"2024-09-13T16:47:59.509298+0000","last_clean":"2024-09-13T16:47:59.509298+0000","last_became_active":"2024-09-13T16:43:54.987048+0000","last_became_peered":"2024-09-13T16:43:54.987048+0000","last_unstale":"2024-09-13T16:47:59.509298+0000","last_undegraded":"2024-09-13T16:47:59.509298+0000","last_fullsized":"2024-09-13T16:47:59.509298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"22'132","reported_seq":145,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.442769+0000","last_change":"2024-09-13T16:43:54.985628+0000","last_active":"2024-09-13T16:47:59.442769+0000","last_peered":"2024-09-13T16:47:59.442769+0000","last_clean":"2024-09-13T16:47:59.442769+0000","last_became_active":"2024-09-13T16:43:54.985378+0000","last_became_peered":"2024-09-13T16:43:54.985378+0000","last_unstale":"2024-09-13T16:47:59.442769+0000","last_undegraded":"2024-09-13T16:47:59.442769+0000","last_fullsized":"2024-09-13T16:47:59.442769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"22'150","reported_seq":163,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.510652+0000","last_change":"2024-09-13T16:43:54.996712+0000","last_active":"2024-09-13T16:47:59.510652+0000","last_peered":"2024-09-13T16:47:59.510652+0000","last_clean":"2024-09-13T16:47:59.510652+0000","last_became_active":"2024-09-13T16:43:54.996523+0000","last_became_peered":"2024-09-13T16:43:54.996523+0000","last_unstale":"2024-09-13T16:47:59.510652+0000","last_undegraded":"2024-09-13T16:47:59.510652+0000","last_fullsized":"2024-09-13T16:47:59.510652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"22'159","reported_seq":172,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.501849+0000","last_change":"2024-09-13T16:43:54.984879+0000","last_active":"2024-09-13T16:47:59.501849+0000","last_peered":"2024-09-13T16:47:59.501849+0000","last_clean":"2024-09-13T16:47:59.501849+0000","last_became_active":"2024-09-13T16:43:54.984595+0000","last_became_peered":"2024-09-13T16:43:54.984595+0000","last_unstale":"2024-09-13T16:47:59.501849+0000","last_undegraded":"2024-09-13T16:47:59.501849+0000","last_fullsized":"2024-09-13T16:47:59.501849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"22'156","reported_seq":169,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.509526+0000","last_change":"2024-09-13T16:43:54.996457+0000","last_active":"2024-09-13T16:47:59.509526+0000","last_peered":"2024-09-13T16:47:59.509526+0000","last_clean":"2024-09-13T16:47:59.509526+0000","last_became_active":"2024-09-13T16:43:54.996275+0000","last_became_peered":"2024-09-13T16:43:54.996275+0000","last_unstale":"2024-09-13T16:47:59.509526+0000","last_undegraded":"2024-09-13T16:47:59.509526+0000","last_fullsized":"2024-09-13T16:47:59.509526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"22'189","reported_seq":202,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.497242+0000","last_change":"2024-09-13T16:43:54.985368+0000","last_active":"2024-09-13T16:47:59.497242+0000","last_peered":"2024-09-13T16:47:59.497242+0000","last_clean":"2024-09-13T16:47:59.497242+0000","last_became_active":"2024-09-13T16:43:54.985130+0000","last_became_peered":"2024-09-13T16:43:54.985130+0000","last_unstale":"2024-09-13T16:47:59.497242+0000","last_undegraded":"2024-09-13T16:47:59.497242+0000","last_fullsized":"2024-09-13T16:47:59.497242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"22'150","reported_seq":163,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.439460+0000","last_change":"2024-09-13T16:43:54.991862+0000","last_active":"2024-09-13T16:47:59.439460+0000","last_peered":"2024-09-13T16:47:59.439460+0000","last_clean":"2024-09-13T16:47:59.439460+0000","last_became_active":"2024-09-13T16:43:54.991554+0000","last_became_peered":"2024-09-13T16:43:54.991554+0000","last_unstale":"2024-09-13T16:47:59.439460+0000","last_undegraded":"2024-09-13T16:47:59.439460+0000","last_fullsized":"2024-09-13T16:47:59.439460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"22'156","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.506350+0000","last_change":"2024-09-13T16:44:59.221091+0000","last_active":"2024-09-13T16:47:59.506350+0000","last_peered":"2024-09-13T16:47:59.506350+0000","last_clean":"2024-09-13T16:47:59.506350+0000","last_became_active":"2024-09-13T16:44:59.220762+0000","last_became_peered":"2024-09-13T16:44:59.220762+0000","last_unstale":"2024-09-13T16:47:59.506350+0000","last_undegraded":"2024-09-13T16:47:59.506350+0000","last_fullsized":"2024-09-13T16:47:59.506350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:39.907088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"22'168","reported_seq":182,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.440760+0000","last_change":"2024-09-13T16:43:54.991366+0000","last_active":"2024-09-13T16:47:59.440760+0000","last_peered":"2024-09-13T16:47:59.440760+0000","last_clean":"2024-09-13T16:47:59.440760+0000","last_became_active":"2024-09-13T16:43:54.991127+0000","last_became_peered":"2024-09-13T16:43:54.991127+0000","last_unstale":"2024-09-13T16:47:59.440760+0000","last_undegraded":"2024-09-13T16:47:59.440760+0000","last_fullsized":"2024-09-13T16:47:59.440760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"22'165","reported_seq":179,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.519235+0000","last_change":"2024-09-13T16:43:54.985615+0000","last_active":"2024-09-13T16:47:59.519235+0000","last_peered":"2024-09-13T16:47:59.519235+0000","last_clean":"2024-09-13T16:47:59.519235+0000","last_became_active":"2024-09-13T16:43:54.985299+0000","last_became_peered":"2024-09-13T16:43:54.985299+0000","last_unstale":"2024-09-13T16:47:59.519235+0000","last_undegraded":"2024-09-13T16:47:59.519235+0000","last_fullsized":"2024-09-13T16:47:59.519235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"22'141","reported_seq":155,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.515984+0000","last_change":"2024-09-13T16:43:54.993325+0000","last_active":"2024-09-13T16:47:59.515984+0000","last_peered":"2024-09-13T16:47:59.515984+0000","last_clean":"2024-09-13T16:47:59.515984+0000","last_became_active":"2024-09-13T16:43:54.993009+0000","last_became_peered":"2024-09-13T16:43:54.993009+0000","last_unstale":"2024-09-13T16:47:59.515984+0000","last_undegraded":"2024-09-13T16:47:59.515984+0000","last_fullsized":"2024-09-13T16:47:59.515984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"22'184","reported_seq":198,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.508028+0000","last_change":"2024-09-13T16:43:54.998229+0000","last_active":"2024-09-13T16:47:59.508028+0000","last_peered":"2024-09-13T16:47:59.508028+0000","last_clean":"2024-09-13T16:47:59.508028+0000","last_became_active":"2024-09-13T16:43:54.995774+0000","last_became_peered":"2024-09-13T16:43:54.995774+0000","last_unstale":"2024-09-13T16:47:59.508028+0000","last_undegraded":"2024-09-13T16:47:59.508028+0000","last_fullsized":"2024-09-13T16:47:59.508028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"22'165","reported_seq":179,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.514352+0000","last_change":"2024-09-13T16:43:55.002126+0000","last_active":"2024-09-13T16:47:59.514352+0000","last_peered":"2024-09-13T16:47:59.514352+0000","last_clean":"2024-09-13T16:47:59.514352+0000","last_became_active":"2024-09-13T16:43:55.001713+0000","last_became_peered":"2024-09-13T16:43:55.001713+0000","last_unstale":"2024-09-13T16:47:59.514352+0000","last_undegraded":"2024-09-13T16:47:59.514352+0000","last_fullsized":"2024-09-13T16:47:59.514352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"22'148","reported_seq":162,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.517055+0000","last_change":"2024-09-13T16:43:54.992745+0000","last_active":"2024-09-13T16:47:59.517055+0000","last_peered":"2024-09-13T16:47:59.517055+0000","last_clean":"2024-09-13T16:47:59.517055+0000","last_became_active":"2024-09-13T16:43:54.992508+0000","last_became_peered":"2024-09-13T16:43:54.992508+0000","last_unstale":"2024-09-13T16:47:59.517055+0000","last_undegraded":"2024-09-13T16:47:59.517055+0000","last_fullsized":"2024-09-13T16:47:59.517055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:23.913470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"22'131","reported_seq":145,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.520706+0000","last_change":"2024-09-13T16:43:54.990631+0000","last_active":"2024-09-13T16:47:59.520706+0000","last_peered":"2024-09-13T16:47:59.520706+0000","last_clean":"2024-09-13T16:47:59.520706+0000","last_became_active":"2024-09-13T16:43:54.986277+0000","last_became_peered":"2024-09-13T16:43:54.986277+0000","last_unstale":"2024-09-13T16:47:59.520706+0000","last_undegraded":"2024-09-13T16:47:59.520706+0000","last_fullsized":"2024-09-13T16:47:59.520706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"22'175","reported_seq":189,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.499494+0000","last_change":"2024-09-13T16:43:55.000059+0000","last_active":"2024-09-13T16:47:59.499494+0000","last_peered":"2024-09-13T16:47:59.499494+0000","last_clean":"2024-09-13T16:47:59.499494+0000","last_became_active":"2024-09-13T16:43:54.999778+0000","last_became_peered":"2024-09-13T16:43:54.999778+0000","last_unstale":"2024-09-13T16:47:59.499494+0000","last_undegraded":"2024-09-13T16:47:59.499494+0000","last_fullsized":"2024-09-13T16:47:59.499494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"22'212","reported_seq":225,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.503806+0000","last_change":"2024-09-13T16:43:54.988375+0000","last_active":"2024-09-13T16:47:59.503806+0000","last_peered":"2024-09-13T16:47:59.503806+0000","last_clean":"2024-09-13T16:47:59.503806+0000","last_became_active":"2024-09-13T16:43:54.988131+0000","last_became_peered":"2024-09-13T16:43:54.988131+0000","last_unstale":"2024-09-13T16:47:59.503806+0000","last_undegraded":"2024-09-13T16:47:59.503806+0000","last_fullsized":"2024-09-13T16:47:59.503806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.635755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.219925+0000","last_change":"2024-09-13T16:43:54.555042+0000","last_active":"2024-09-13T16:44:59.219925+0000","last_peered":"2024-09-13T16:44:59.219925+0000","last_clean":"2024-09-13T16:44:59.219925+0000","last_became_active":"2024-09-13T16:43:54.554697+0000","last_became_peered":"2024-09-13T16:43:54.554697+0000","last_unstale":"2024-09-13T16:44:59.219925+0000","last_undegraded":"2024-09-13T16:44:59.219925+0000","last_fullsized":"2024-09-13T16:44:59.219925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"22'165","reported_seq":178,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.517461+0000","last_change":"2024-09-13T16:43:54.998755+0000","last_active":"2024-09-13T16:47:59.517461+0000","last_peered":"2024-09-13T16:47:59.517461+0000","last_clean":"2024-09-13T16:47:59.517461+0000","last_became_active":"2024-09-13T16:43:54.998399+0000","last_became_peered":"2024-09-13T16:43:54.998399+0000","last_unstale":"2024-09-13T16:47:59.517461+0000","last_undegraded":"2024-09-13T16:47:59.517461+0000","last_fullsized":"2024-09-13T16:47:59.517461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.215854+0000","last_change":"2024-09-13T16:43:54.553250+0000","last_active":"2024-09-13T16:44:59.215854+0000","last_peered":"2024-09-13T16:44:59.215854+0000","last_clean":"2024-09-13T16:44:59.215854+0000","last_became_active":"2024-09-13T16:43:54.552977+0000","last_became_peered":"2024-09-13T16:43:54.552977+0000","last_unstale":"2024-09-13T16:44:59.215854+0000","last_undegraded":"2024-09-13T16:44:59.215854+0000","last_fullsized":"2024-09-13T16:44:59.215854+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.219531+0000","last_change":"2024-09-13T16:43:54.558065+0000","last_active":"2024-09-13T16:44:59.219531+0000","last_peered":"2024-09-13T16:44:59.219531+0000","last_clean":"2024-09-13T16:44:59.219531+0000","last_became_active":"2024-09-13T16:43:54.557722+0000","last_became_peered":"2024-09-13T16:43:54.557722+0000","last_unstale":"2024-09-13T16:44:59.219531+0000","last_undegraded":"2024-09-13T16:44:59.219531+0000","last_fullsized":"2024-09-13T16:44:59.219531+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"22'151","reported_seq":164,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.495858+0000","last_change":"2024-09-13T16:43:55.002041+0000","last_active":"2024-09-13T16:47:59.495858+0000","last_peered":"2024-09-13T16:47:59.495858+0000","last_clean":"2024-09-13T16:47:59.495858+0000","last_became_active":"2024-09-13T16:43:55.001512+0000","last_became_peered":"2024-09-13T16:43:55.001512+0000","last_unstale":"2024-09-13T16:47:59.495858+0000","last_undegraded":"2024-09-13T16:47:59.495858+0000","last_fullsized":"2024-09-13T16:47:59.495858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.383328+0000","last_change":"2024-09-13T16:43:54.550389+0000","last_active":"2024-09-13T16:44:59.383328+0000","last_peered":"2024-09-13T16:44:59.383328+0000","last_clean":"2024-09-13T16:44:59.383328+0000","last_became_active":"2024-09-13T16:43:54.550154+0000","last_became_peered":"2024-09-13T16:43:54.550154+0000","last_unstale":"2024-09-13T16:44:59.383328+0000","last_undegraded":"2024-09-13T16:44:59.383328+0000","last_fullsized":"2024-09-13T16:44:59.383328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"22'168","reported_seq":181,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.449760+0000","last_change":"2024-09-13T16:43:54.993377+0000","last_active":"2024-09-13T16:47:59.449760+0000","last_peered":"2024-09-13T16:47:59.449760+0000","last_clean":"2024-09-13T16:47:59.449760+0000","last_became_active":"2024-09-13T16:43:54.993036+0000","last_became_peered":"2024-09-13T16:43:54.993036+0000","last_unstale":"2024-09-13T16:47:59.449760+0000","last_undegraded":"2024-09-13T16:47:59.449760+0000","last_fullsized":"2024-09-13T16:47:59.449760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:13.068438+0000","last_change":"2024-09-13T16:43:54.555528+0000","last_active":"2024-09-13T16:46:13.068438+0000","last_peered":"2024-09-13T16:46:13.068438+0000","last_clean":"2024-09-13T16:46:13.068438+0000","last_became_active":"2024-09-13T16:43:54.555267+0000","last_became_peered":"2024-09-13T16:43:54.555267+0000","last_unstale":"2024-09-13T16:46:13.068438+0000","last_undegraded":"2024-09-13T16:46:13.068438+0000","last_fullsized":"2024-09-13T16:46:13.068438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"22'143","reported_seq":156,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.438299+0000","last_change":"2024-09-13T16:43:54.981706+0000","last_active":"2024-09-13T16:47:59.438299+0000","last_peered":"2024-09-13T16:47:59.438299+0000","last_clean":"2024-09-13T16:47:59.438299+0000","last_became_active":"2024-09-13T16:43:54.981429+0000","last_became_peered":"2024-09-13T16:43:54.981429+0000","last_unstale":"2024-09-13T16:47:59.438299+0000","last_undegraded":"2024-09-13T16:47:59.438299+0000","last_fullsized":"2024-09-13T16:47:59.438299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.382875+0000","last_change":"2024-09-13T16:43:54.554593+0000","last_active":"2024-09-13T16:44:59.382875+0000","last_peered":"2024-09-13T16:44:59.382875+0000","last_clean":"2024-09-13T16:44:59.382875+0000","last_became_active":"2024-09-13T16:43:54.554307+0000","last_became_peered":"2024-09-13T16:43:54.554307+0000","last_unstale":"2024-09-13T16:44:59.382875+0000","last_undegraded":"2024-09-13T16:44:59.382875+0000","last_fullsized":"2024-09-13T16:44:59.382875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"22'195","reported_seq":208,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.497022+0000","last_change":"2024-09-13T16:43:54.993485+0000","last_active":"2024-09-13T16:47:59.497022+0000","last_peered":"2024-09-13T16:47:59.497022+0000","last_clean":"2024-09-13T16:47:59.497022+0000","last_became_active":"2024-09-13T16:43:54.993226+0000","last_became_peered":"2024-09-13T16:43:54.993226+0000","last_unstale":"2024-09-13T16:47:59.497022+0000","last_undegraded":"2024-09-13T16:47:59.497022+0000","last_fullsized":"2024-09-13T16:47:59.497022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'11","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:03.123692+0000","last_change":"2024-09-13T16:43:54.562976+0000","last_active":"2024-09-13T16:48:03.123692+0000","last_peered":"2024-09-13T16:48:03.123692+0000","last_clean":"2024-09-13T16:48:03.123692+0000","last_became_active":"2024-09-13T16:43:54.560480+0000","last_became_peered":"2024-09-13T16:43:54.560480+0000","last_unstale":"2024-09-13T16:48:03.123692+0000","last_undegraded":"2024-09-13T16:48:03.123692+0000","last_fullsized":"2024-09-13T16:48:03.123692+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"22'200","reported_seq":213,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.501250+0000","last_change":"2024-09-13T16:43:54.986459+0000","last_active":"2024-09-13T16:47:59.501250+0000","last_peered":"2024-09-13T16:47:59.501250+0000","last_clean":"2024-09-13T16:47:59.501250+0000","last_became_active":"2024-09-13T16:43:54.986083+0000","last_became_peered":"2024-09-13T16:43:54.986083+0000","last_unstale":"2024-09-13T16:47:59.501250+0000","last_undegraded":"2024-09-13T16:47:59.501250+0000","last_fullsized":"2024-09-13T16:47:59.501250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:51.960459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'165","reported_seq":179,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:14.014679+0000","last_change":"2024-09-13T16:43:54.565929+0000","last_active":"2024-09-13T16:48:14.014679+0000","last_peered":"2024-09-13T16:48:14.014679+0000","last_clean":"2024-09-13T16:48:14.014679+0000","last_became_active":"2024-09-13T16:43:54.565701+0000","last_became_peered":"2024-09-13T16:43:54.565701+0000","last_unstale":"2024-09-13T16:48:14.014679+0000","last_undegraded":"2024-09-13T16:48:14.014679+0000","last_fullsized":"2024-09-13T16:48:14.014679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364449,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":166,"num_write_kb":479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"22'235","reported_seq":248,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.515163+0000","last_change":"2024-09-13T16:43:54.989068+0000","last_active":"2024-09-13T16:47:59.515163+0000","last_peered":"2024-09-13T16:47:59.515163+0000","last_clean":"2024-09-13T16:47:59.515163+0000","last_became_active":"2024-09-13T16:43:54.988779+0000","last_became_peered":"2024-09-13T16:43:54.988779+0000","last_unstale":"2024-09-13T16:47:59.515163+0000","last_undegraded":"2024-09-13T16:47:59.515163+0000","last_fullsized":"2024-09-13T16:47:59.515163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":213,"num_write_kb":581632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.138534+0000","last_change":"2024-09-13T16:43:54.561487+0000","last_active":"2024-09-13T16:44:59.138534+0000","last_peered":"2024-09-13T16:44:59.138534+0000","last_clean":"2024-09-13T16:44:59.138534+0000","last_became_active":"2024-09-13T16:43:54.561077+0000","last_became_peered":"2024-09-13T16:43:54.561077+0000","last_unstale":"2024-09-13T16:44:59.138534+0000","last_undegraded":"2024-09-13T16:44:59.138534+0000","last_fullsized":"2024-09-13T16:44:59.138534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"22'153","reported_seq":166,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.500527+0000","last_change":"2024-09-13T16:43:54.999517+0000","last_active":"2024-09-13T16:47:59.500527+0000","last_peered":"2024-09-13T16:47:59.500527+0000","last_clean":"2024-09-13T16:47:59.500527+0000","last_became_active":"2024-09-13T16:43:54.999283+0000","last_became_peered":"2024-09-13T16:43:54.999283+0000","last_unstale":"2024-09-13T16:47:59.500527+0000","last_undegraded":"2024-09-13T16:47:59.500527+0000","last_fullsized":"2024-09-13T16:47:59.500527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:20.207775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.383950+0000","last_change":"2024-09-13T16:43:54.542871+0000","last_active":"2024-09-13T16:44:59.383950+0000","last_peered":"2024-09-13T16:44:59.383950+0000","last_clean":"2024-09-13T16:44:59.383950+0000","last_became_active":"2024-09-13T16:43:54.542548+0000","last_became_peered":"2024-09-13T16:43:54.542548+0000","last_unstale":"2024-09-13T16:44:59.383950+0000","last_undegraded":"2024-09-13T16:44:59.383950+0000","last_fullsized":"2024-09-13T16:44:59.383950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"22'180","reported_seq":193,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.452606+0000","last_change":"2024-09-13T16:43:54.989168+0000","last_active":"2024-09-13T16:47:59.452606+0000","last_peered":"2024-09-13T16:47:59.452606+0000","last_clean":"2024-09-13T16:47:59.452606+0000","last_became_active":"2024-09-13T16:43:54.988716+0000","last_became_peered":"2024-09-13T16:43:54.988716+0000","last_unstale":"2024-09-13T16:47:59.452606+0000","last_undegraded":"2024-09-13T16:47:59.452606+0000","last_fullsized":"2024-09-13T16:47:59.452606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"20'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.215766+0000","last_change":"2024-09-13T16:43:54.551551+0000","last_active":"2024-09-13T16:44:59.215766+0000","last_peered":"2024-09-13T16:44:59.215766+0000","last_clean":"2024-09-13T16:44:59.215766+0000","last_became_active":"2024-09-13T16:43:54.551293+0000","last_became_peered":"2024-09-13T16:43:54.551293+0000","last_unstale":"2024-09-13T16:44:59.215766+0000","last_undegraded":"2024-09-13T16:44:59.215766+0000","last_fullsized":"2024-09-13T16:44:59.215766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":799,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"22'163","reported_seq":176,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.499067+0000","last_change":"2024-09-13T16:43:54.994516+0000","last_active":"2024-09-13T16:47:59.499067+0000","last_peered":"2024-09-13T16:47:59.499067+0000","last_clean":"2024-09-13T16:47:59.499067+0000","last_became_active":"2024-09-13T16:43:54.994245+0000","last_became_peered":"2024-09-13T16:43:54.994245+0000","last_unstale":"2024-09-13T16:47:59.499067+0000","last_undegraded":"2024-09-13T16:47:59.499067+0000","last_fullsized":"2024-09-13T16:47:59.499067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'18","reported_seq":32,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:14.507552+0000","last_change":"2024-09-13T16:43:54.562232+0000","last_active":"2024-09-13T16:46:14.507552+0000","last_peered":"2024-09-13T16:46:14.507552+0000","last_clean":"2024-09-13T16:46:14.507552+0000","last_became_active":"2024-09-13T16:43:54.561939+0000","last_became_peered":"2024-09-13T16:43:54.561939+0000","last_unstale":"2024-09-13T16:46:14.507552+0000","last_undegraded":"2024-09-13T16:46:14.507552+0000","last_fullsized":"2024-09-13T16:46:14.507552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"22'169","reported_seq":182,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.452403+0000","last_change":"2024-09-13T16:43:54.988439+0000","last_active":"2024-09-13T16:47:59.452403+0000","last_peered":"2024-09-13T16:47:59.452403+0000","last_clean":"2024-09-13T16:47:59.452403+0000","last_became_active":"2024-09-13T16:43:54.988172+0000","last_became_peered":"2024-09-13T16:43:54.988172+0000","last_unstale":"2024-09-13T16:47:59.452403+0000","last_undegraded":"2024-09-13T16:47:59.452403+0000","last_fullsized":"2024-09-13T16:47:59.452403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:58.954977+0000","last_change":"2024-09-13T16:43:54.559372+0000","last_active":"2024-09-13T16:44:58.954977+0000","last_peered":"2024-09-13T16:44:58.954977+0000","last_clean":"2024-09-13T16:44:58.954977+0000","last_became_active":"2024-09-13T16:43:54.559111+0000","last_became_peered":"2024-09-13T16:43:54.559111+0000","last_unstale":"2024-09-13T16:44:58.954977+0000","last_undegraded":"2024-09-13T16:44:58.954977+0000","last_fullsized":"2024-09-13T16:44:58.954977+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:40.399883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'151","reported_seq":164,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.509322+0000","last_change":"2024-09-13T16:43:54.988313+0000","last_active":"2024-09-13T16:47:59.509322+0000","last_peered":"2024-09-13T16:47:59.509322+0000","last_clean":"2024-09-13T16:47:59.509322+0000","last_became_active":"2024-09-13T16:43:54.988059+0000","last_became_peered":"2024-09-13T16:43:54.988059+0000","last_unstale":"2024-09-13T16:47:59.509322+0000","last_undegraded":"2024-09-13T16:47:59.509322+0000","last_fullsized":"2024-09-13T16:47:59.509322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:58.955244+0000","last_change":"2024-09-13T16:43:54.566410+0000","last_active":"2024-09-13T16:44:58.955244+0000","last_peered":"2024-09-13T16:44:58.955244+0000","last_clean":"2024-09-13T16:44:58.955244+0000","last_became_active":"2024-09-13T16:43:54.565861+0000","last_became_peered":"2024-09-13T16:43:54.565861+0000","last_unstale":"2024-09-13T16:44:58.955244+0000","last_undegraded":"2024-09-13T16:44:58.955244+0000","last_fullsized":"2024-09-13T16:44:58.955244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"22'127","reported_seq":138,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.428730+0000","last_change":"2024-09-13T16:44:58.956863+0000","last_active":"2024-09-13T16:47:59.428730+0000","last_peered":"2024-09-13T16:47:59.428730+0000","last_clean":"2024-09-13T16:47:59.428730+0000","last_became_active":"2024-09-13T16:44:58.956557+0000","last_became_peered":"2024-09-13T16:44:58.956557+0000","last_unstale":"2024-09-13T16:47:59.428730+0000","last_undegraded":"2024-09-13T16:47:59.428730+0000","last_fullsized":"2024-09-13T16:47:59.428730+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:10.217411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":278528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:58.957395+0000","last_change":"2024-09-13T16:43:54.559723+0000","last_active":"2024-09-13T16:44:58.957395+0000","last_peered":"2024-09-13T16:44:58.957395+0000","last_clean":"2024-09-13T16:44:58.957395+0000","last_became_active":"2024-09-13T16:43:54.559353+0000","last_became_peered":"2024-09-13T16:43:54.559353+0000","last_unstale":"2024-09-13T16:44:58.957395+0000","last_undegraded":"2024-09-13T16:44:58.957395+0000","last_fullsized":"2024-09-13T16:44:58.957395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"22'203","reported_seq":216,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.498887+0000","last_change":"2024-09-13T16:43:54.992511+0000","last_active":"2024-09-13T16:47:59.498887+0000","last_peered":"2024-09-13T16:47:59.498887+0000","last_clean":"2024-09-13T16:47:59.498887+0000","last_became_active":"2024-09-13T16:43:54.992249+0000","last_became_peered":"2024-09-13T16:43:54.992249+0000","last_unstale":"2024-09-13T16:47:59.498887+0000","last_undegraded":"2024-09-13T16:47:59.498887+0000","last_fullsized":"2024-09-13T16:47:59.498887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"20'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:58.955067+0000","last_change":"2024-09-13T16:43:54.551390+0000","last_active":"2024-09-13T16:44:58.955067+0000","last_peered":"2024-09-13T16:44:58.955067+0000","last_clean":"2024-09-13T16:44:58.955067+0000","last_became_active":"2024-09-13T16:43:54.547074+0000","last_became_peered":"2024-09-13T16:43:54.547074+0000","last_unstale":"2024-09-13T16:44:58.955067+0000","last_undegraded":"2024-09-13T16:44:58.955067+0000","last_fullsized":"2024-09-13T16:44:58.955067+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"22'165","reported_seq":178,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.492614+0000","last_change":"2024-09-13T16:43:54.990458+0000","last_active":"2024-09-13T16:47:59.492614+0000","last_peered":"2024-09-13T16:47:59.492614+0000","last_clean":"2024-09-13T16:47:59.492614+0000","last_became_active":"2024-09-13T16:43:54.990196+0000","last_became_peered":"2024-09-13T16:43:54.990196+0000","last_unstale":"2024-09-13T16:47:59.492614+0000","last_undegraded":"2024-09-13T16:47:59.492614+0000","last_fullsized":"2024-09-13T16:47:59.492614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:54.505478+0000","last_change":"2024-09-13T16:43:51.914639+0000","last_active":"2024-09-13T16:45:54.505478+0000","last_peered":"2024-09-13T16:45:54.505478+0000","last_clean":"2024-09-13T16:45:54.505478+0000","last_became_active":"2024-09-13T16:43:51.913797+0000","last_became_peered":"2024-09-13T16:43:51.913797+0000","last_unstale":"2024-09-13T16:45:54.505478+0000","last_undegraded":"2024-09-13T16:45:54.505478+0000","last_fullsized":"2024-09-13T16:45:54.505478+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_clean_scrub_stamp":"2024-09-13T16:43:50.498116+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:12.730911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:00.501470+0000","last_change":"2024-09-13T16:43:54.564153+0000","last_active":"2024-09-13T16:46:00.501470+0000","last_peered":"2024-09-13T16:46:00.501470+0000","last_clean":"2024-09-13T16:46:00.501470+0000","last_became_active":"2024-09-13T16:43:54.563949+0000","last_became_peered":"2024-09-13T16:43:54.563949+0000","last_unstale":"2024-09-13T16:46:00.501470+0000","last_undegraded":"2024-09-13T16:46:00.501470+0000","last_fullsized":"2024-09-13T16:46:00.501470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"22'207","reported_seq":218,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.437738+0000","last_change":"2024-09-13T16:44:59.387015+0000","last_active":"2024-09-13T16:47:59.437738+0000","last_peered":"2024-09-13T16:47:59.437738+0000","last_clean":"2024-09-13T16:47:59.437738+0000","last_became_active":"2024-09-13T16:44:59.386756+0000","last_became_peered":"2024-09-13T16:44:59.386756+0000","last_unstale":"2024-09-13T16:47:59.437738+0000","last_undegraded":"2024-09-13T16:47:59.437738+0000","last_fullsized":"2024-09-13T16:47:59.437738+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:15.100276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":186,"num_write_kb":507904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:54.505444+0000","last_change":"2024-09-13T16:43:54.564804+0000","last_active":"2024-09-13T16:45:54.505444+0000","last_peered":"2024-09-13T16:45:54.505444+0000","last_clean":"2024-09-13T16:45:54.505444+0000","last_became_active":"2024-09-13T16:43:54.564605+0000","last_became_peered":"2024-09-13T16:43:54.564605+0000","last_unstale":"2024-09-13T16:45:54.505444+0000","last_undegraded":"2024-09-13T16:45:54.505444+0000","last_fullsized":"2024-09-13T16:45:54.505444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"22'187","reported_seq":201,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.514834+0000","last_change":"2024-09-13T16:43:54.997884+0000","last_active":"2024-09-13T16:47:59.514834+0000","last_peered":"2024-09-13T16:47:59.514834+0000","last_clean":"2024-09-13T16:47:59.514834+0000","last_became_active":"2024-09-13T16:43:54.997510+0000","last_became_peered":"2024-09-13T16:43:54.997510+0000","last_unstale":"2024-09-13T16:47:59.514834+0000","last_undegraded":"2024-09-13T16:47:59.514834+0000","last_fullsized":"2024-09-13T16:47:59.514834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'12","reported_seq":32,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:10.123144+0000","last_change":"2024-09-13T16:43:54.566294+0000","last_active":"2024-09-13T16:48:10.123144+0000","last_peered":"2024-09-13T16:48:10.123144+0000","last_clean":"2024-09-13T16:48:10.123144+0000","last_became_active":"2024-09-13T16:43:54.566023+0000","last_became_peered":"2024-09-13T16:43:54.566023+0000","last_unstale":"2024-09-13T16:48:10.123144+0000","last_undegraded":"2024-09-13T16:48:10.123144+0000","last_fullsized":"2024-09-13T16:48:10.123144+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"22'159","reported_seq":173,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.518351+0000","last_change":"2024-09-13T16:43:54.981116+0000","last_active":"2024-09-13T16:47:59.518351+0000","last_peered":"2024-09-13T16:47:59.518351+0000","last_clean":"2024-09-13T16:47:59.518351+0000","last_became_active":"2024-09-13T16:43:54.980766+0000","last_became_peered":"2024-09-13T16:43:54.980766+0000","last_unstale":"2024-09-13T16:47:59.518351+0000","last_undegraded":"2024-09-13T16:47:59.518351+0000","last_fullsized":"2024-09-13T16:47:59.518351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:00.501452+0000","last_change":"2024-09-13T16:43:54.563716+0000","last_active":"2024-09-13T16:46:00.501452+0000","last_peered":"2024-09-13T16:46:00.501452+0000","last_clean":"2024-09-13T16:46:00.501452+0000","last_became_active":"2024-09-13T16:43:54.563522+0000","last_became_peered":"2024-09-13T16:43:54.563522+0000","last_unstale":"2024-09-13T16:46:00.501452+0000","last_undegraded":"2024-09-13T16:46:00.501452+0000","last_fullsized":"2024-09-13T16:46:00.501452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"22'156","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.513433+0000","last_change":"2024-09-13T16:43:54.999051+0000","last_active":"2024-09-13T16:47:59.513433+0000","last_peered":"2024-09-13T16:47:59.513433+0000","last_clean":"2024-09-13T16:47:59.513433+0000","last_became_active":"2024-09-13T16:43:54.998803+0000","last_became_peered":"2024-09-13T16:43:54.998803+0000","last_unstale":"2024-09-13T16:47:59.513433+0000","last_undegraded":"2024-09-13T16:47:59.513433+0000","last_fullsized":"2024-09-13T16:47:59.513433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:59.505325+0000","last_change":"2024-09-13T16:43:54.569945+0000","last_active":"2024-09-13T16:45:59.505325+0000","last_peered":"2024-09-13T16:45:59.505325+0000","last_clean":"2024-09-13T16:45:59.505325+0000","last_became_active":"2024-09-13T16:43:54.568846+0000","last_became_peered":"2024-09-13T16:43:54.568846+0000","last_unstale":"2024-09-13T16:45:59.505325+0000","last_undegraded":"2024-09-13T16:45:59.505325+0000","last_fullsized":"2024-09-13T16:45:59.505325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"22'175","reported_seq":189,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.514055+0000","last_change":"2024-09-13T16:43:54.992082+0000","last_active":"2024-09-13T16:47:59.514055+0000","last_peered":"2024-09-13T16:47:59.514055+0000","last_clean":"2024-09-13T16:47:59.514055+0000","last_became_active":"2024-09-13T16:43:54.991659+0000","last_became_peered":"2024-09-13T16:43:54.991659+0000","last_unstale":"2024-09-13T16:47:59.514055+0000","last_undegraded":"2024-09-13T16:47:59.514055+0000","last_fullsized":"2024-09-13T16:47:59.514055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:59.506045+0000","last_change":"2024-09-13T16:43:54.569111+0000","last_active":"2024-09-13T16:45:59.506045+0000","last_peered":"2024-09-13T16:45:59.506045+0000","last_clean":"2024-09-13T16:45:59.506045+0000","last_became_active":"2024-09-13T16:43:54.568903+0000","last_became_peered":"2024-09-13T16:43:54.568903+0000","last_unstale":"2024-09-13T16:45:59.506045+0000","last_undegraded":"2024-09-13T16:45:59.506045+0000","last_fullsized":"2024-09-13T16:45:59.506045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"22'197","reported_seq":211,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.518940+0000","last_change":"2024-09-13T16:43:54.980100+0000","last_active":"2024-09-13T16:47:59.518940+0000","last_peered":"2024-09-13T16:47:59.518940+0000","last_clean":"2024-09-13T16:47:59.518940+0000","last_became_active":"2024-09-13T16:43:54.979584+0000","last_became_peered":"2024-09-13T16:43:54.979584+0000","last_unstale":"2024-09-13T16:47:59.518940+0000","last_undegraded":"2024-09-13T16:47:59.518940+0000","last_fullsized":"2024-09-13T16:47:59.518940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'20","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:14.014651+0000","last_change":"2024-09-13T16:43:54.560710+0000","last_active":"2024-09-13T16:48:14.014651+0000","last_peered":"2024-09-13T16:48:14.014651+0000","last_clean":"2024-09-13T16:48:14.014651+0000","last_became_active":"2024-09-13T16:43:54.560030+0000","last_became_peered":"2024-09-13T16:43:54.560030+0000","last_unstale":"2024-09-13T16:48:14.014651+0000","last_undegraded":"2024-09-13T16:48:14.014651+0000","last_fullsized":"2024-09-13T16:48:14.014651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'191","reported_seq":205,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.510771+0000","last_change":"2024-09-13T16:43:54.990812+0000","last_active":"2024-09-13T16:47:59.510771+0000","last_peered":"2024-09-13T16:47:59.510771+0000","last_clean":"2024-09-13T16:47:59.510771+0000","last_became_active":"2024-09-13T16:43:54.990440+0000","last_became_peered":"2024-09-13T16:43:54.990440+0000","last_unstale":"2024-09-13T16:47:59.510771+0000","last_undegraded":"2024-09-13T16:47:59.510771+0000","last_fullsized":"2024-09-13T16:47:59.510771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'8","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:03.122078+0000","last_change":"2024-09-13T16:43:54.558439+0000","last_active":"2024-09-13T16:48:03.122078+0000","last_peered":"2024-09-13T16:48:03.122078+0000","last_clean":"2024-09-13T16:48:03.122078+0000","last_became_active":"2024-09-13T16:43:54.558194+0000","last_became_peered":"2024-09-13T16:43:54.558194+0000","last_unstale":"2024-09-13T16:48:03.122078+0000","last_undegraded":"2024-09-13T16:48:03.122078+0000","last_fullsized":"2024-09-13T16:48:03.122078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"22'154","reported_seq":168,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.507374+0000","last_change":"2024-09-13T16:43:54.995534+0000","last_active":"2024-09-13T16:47:59.507374+0000","last_peered":"2024-09-13T16:47:59.507374+0000","last_clean":"2024-09-13T16:47:59.507374+0000","last_became_active":"2024-09-13T16:43:54.995397+0000","last_became_peered":"2024-09-13T16:43:54.995397+0000","last_unstale":"2024-09-13T16:47:59.507374+0000","last_undegraded":"2024-09-13T16:47:59.507374+0000","last_fullsized":"2024-09-13T16:47:59.507374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:19.697247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:04.506432+0000","last_change":"2024-09-13T16:43:54.569085+0000","last_active":"2024-09-13T16:46:04.506432+0000","last_peered":"2024-09-13T16:46:04.506432+0000","last_clean":"2024-09-13T16:46:04.506432+0000","last_became_active":"2024-09-13T16:43:54.568892+0000","last_became_peered":"2024-09-13T16:43:54.568892+0000","last_unstale":"2024-09-13T16:46:04.506432+0000","last_undegraded":"2024-09-13T16:46:04.506432+0000","last_fullsized":"2024-09-13T16:46:04.506432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"22'196","reported_seq":210,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.498566+0000","last_change":"2024-09-13T16:43:54.984120+0000","last_active":"2024-09-13T16:47:59.498566+0000","last_peered":"2024-09-13T16:47:59.498566+0000","last_clean":"2024-09-13T16:47:59.498566+0000","last_became_active":"2024-09-13T16:43:54.983886+0000","last_became_peered":"2024-09-13T16:43:54.983886+0000","last_unstale":"2024-09-13T16:47:59.498566+0000","last_undegraded":"2024-09-13T16:47:59.498566+0000","last_fullsized":"2024-09-13T16:47:59.498566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:04.505849+0000","last_change":"2024-09-13T16:43:54.555092+0000","last_active":"2024-09-13T16:46:04.505849+0000","last_peered":"2024-09-13T16:46:04.505849+0000","last_clean":"2024-09-13T16:46:04.505849+0000","last_became_active":"2024-09-13T16:43:54.554230+0000","last_became_peered":"2024-09-13T16:43:54.554230+0000","last_unstale":"2024-09-13T16:46:04.505849+0000","last_undegraded":"2024-09-13T16:46:04.505849+0000","last_fullsized":"2024-09-13T16:46:04.505849+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"22'169","reported_seq":183,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.513275+0000","last_change":"2024-09-13T16:43:54.984588+0000","last_active":"2024-09-13T16:47:59.513275+0000","last_peered":"2024-09-13T16:47:59.513275+0000","last_clean":"2024-09-13T16:47:59.513275+0000","last_became_active":"2024-09-13T16:43:54.984221+0000","last_became_peered":"2024-09-13T16:43:54.984221+0000","last_unstale":"2024-09-13T16:47:59.513275+0000","last_undegraded":"2024-09-13T16:47:59.513275+0000","last_fullsized":"2024-09-13T16:47:59.513275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:04.505864+0000","last_change":"2024-09-13T16:43:54.562646+0000","last_active":"2024-09-13T16:46:04.505864+0000","last_peered":"2024-09-13T16:46:04.505864+0000","last_clean":"2024-09-13T16:46:04.505864+0000","last_became_active":"2024-09-13T16:43:54.556888+0000","last_became_peered":"2024-09-13T16:43:54.556888+0000","last_unstale":"2024-09-13T16:46:04.505864+0000","last_undegraded":"2024-09-13T16:46:04.505864+0000","last_fullsized":"2024-09-13T16:46:04.505864+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"22'209","reported_seq":222,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.429450+0000","last_change":"2024-09-13T16:43:54.999328+0000","last_active":"2024-09-13T16:47:59.429450+0000","last_peered":"2024-09-13T16:47:59.429450+0000","last_clean":"2024-09-13T16:47:59.429450+0000","last_became_active":"2024-09-13T16:43:54.999101+0000","last_became_peered":"2024-09-13T16:43:54.999101+0000","last_unstale":"2024-09-13T16:47:59.429450+0000","last_undegraded":"2024-09-13T16:47:59.429450+0000","last_fullsized":"2024-09-13T16:47:59.429450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:09.507033+0000","last_change":"2024-09-13T16:43:54.559214+0000","last_active":"2024-09-13T16:46:09.507033+0000","last_peered":"2024-09-13T16:46:09.507033+0000","last_clean":"2024-09-13T16:46:09.507033+0000","last_became_active":"2024-09-13T16:43:54.558233+0000","last_became_peered":"2024-09-13T16:43:54.558233+0000","last_unstale":"2024-09-13T16:46:09.507033+0000","last_undegraded":"2024-09-13T16:46:09.507033+0000","last_fullsized":"2024-09-13T16:46:09.507033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"22'185","reported_seq":199,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.519284+0000","last_change":"2024-09-13T16:43:54.986565+0000","last_active":"2024-09-13T16:47:59.519284+0000","last_peered":"2024-09-13T16:47:59.519284+0000","last_clean":"2024-09-13T16:47:59.519284+0000","last_became_active":"2024-09-13T16:43:54.986084+0000","last_became_peered":"2024-09-13T16:43:54.986084+0000","last_unstale":"2024-09-13T16:47:59.519284+0000","last_undegraded":"2024-09-13T16:47:59.519284+0000","last_fullsized":"2024-09-13T16:47:59.519284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"22'150","reported_seq":163,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.519001+0000","last_change":"2024-09-13T16:43:54.982611+0000","last_active":"2024-09-13T16:47:59.519001+0000","last_peered":"2024-09-13T16:47:59.519001+0000","last_clean":"2024-09-13T16:47:59.519001+0000","last_became_active":"2024-09-13T16:43:54.982338+0000","last_became_peered":"2024-09-13T16:43:54.982338+0000","last_unstale":"2024-09-13T16:47:59.519001+0000","last_undegraded":"2024-09-13T16:47:59.519001+0000","last_fullsized":"2024-09-13T16:47:59.519001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.382688+0000","last_change":"2024-09-13T16:43:54.558799+0000","last_active":"2024-09-13T16:44:59.382688+0000","last_peered":"2024-09-13T16:44:59.382688+0000","last_clean":"2024-09-13T16:44:59.382688+0000","last_became_active":"2024-09-13T16:43:54.558332+0000","last_became_peered":"2024-09-13T16:43:54.558332+0000","last_unstale":"2024-09-13T16:44:59.382688+0000","last_undegraded":"2024-09-13T16:44:59.382688+0000","last_fullsized":"2024-09-13T16:44:59.382688+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:19.201506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'143","reported_seq":154,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.415246+0000","last_change":"2024-09-13T16:44:58.957875+0000","last_active":"2024-09-13T16:47:59.415246+0000","last_peered":"2024-09-13T16:47:59.415246+0000","last_clean":"2024-09-13T16:47:59.415246+0000","last_became_active":"2024-09-13T16:44:58.957568+0000","last_became_peered":"2024-09-13T16:44:58.957568+0000","last_unstale":"2024-09-13T16:47:59.415246+0000","last_undegraded":"2024-09-13T16:47:59.415246+0000","last_fullsized":"2024-09-13T16:47:59.415246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:28.185704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'13","reported_seq":27,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:13.069441+0000","last_change":"2024-09-13T16:43:54.559551+0000","last_active":"2024-09-13T16:46:13.069441+0000","last_peered":"2024-09-13T16:46:13.069441+0000","last_clean":"2024-09-13T16:46:13.069441+0000","last_became_active":"2024-09-13T16:43:54.559290+0000","last_became_peered":"2024-09-13T16:43:54.559290+0000","last_unstale":"2024-09-13T16:46:13.069441+0000","last_undegraded":"2024-09-13T16:46:13.069441+0000","last_fullsized":"2024-09-13T16:46:13.069441+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"22'174","reported_seq":188,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.508073+0000","last_change":"2024-09-13T16:43:55.000783+0000","last_active":"2024-09-13T16:47:59.508073+0000","last_peered":"2024-09-13T16:47:59.508073+0000","last_clean":"2024-09-13T16:47:59.508073+0000","last_became_active":"2024-09-13T16:43:55.000196+0000","last_became_peered":"2024-09-13T16:43:55.000196+0000","last_unstale":"2024-09-13T16:47:59.508073+0000","last_undegraded":"2024-09-13T16:47:59.508073+0000","last_fullsized":"2024-09-13T16:47:59.508073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:54.504808+0000","last_change":"2024-09-13T16:43:54.559886+0000","last_active":"2024-09-13T16:45:54.504808+0000","last_peered":"2024-09-13T16:45:54.504808+0000","last_clean":"2024-09-13T16:45:54.504808+0000","last_became_active":"2024-09-13T16:43:54.559586+0000","last_became_peered":"2024-09-13T16:43:54.559586+0000","last_unstale":"2024-09-13T16:45:54.504808+0000","last_undegraded":"2024-09-13T16:45:54.504808+0000","last_fullsized":"2024-09-13T16:45:54.504808+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"22'147","reported_seq":161,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.520197+0000","last_change":"2024-09-13T16:43:54.996099+0000","last_active":"2024-09-13T16:47:59.520197+0000","last_peered":"2024-09-13T16:47:59.520197+0000","last_clean":"2024-09-13T16:47:59.520197+0000","last_became_active":"2024-09-13T16:43:54.995802+0000","last_became_peered":"2024-09-13T16:43:54.995802+0000","last_unstale":"2024-09-13T16:47:59.520197+0000","last_undegraded":"2024-09-13T16:47:59.520197+0000","last_fullsized":"2024-09-13T16:47:59.520197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":28,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:59.505735+0000","last_change":"2024-09-13T16:43:54.562699+0000","last_active":"2024-09-13T16:45:59.505735+0000","last_peered":"2024-09-13T16:45:59.505735+0000","last_clean":"2024-09-13T16:45:59.505735+0000","last_became_active":"2024-09-13T16:43:54.560552+0000","last_became_peered":"2024-09-13T16:43:54.560552+0000","last_unstale":"2024-09-13T16:45:59.505735+0000","last_undegraded":"2024-09-13T16:45:59.505735+0000","last_fullsized":"2024-09-13T16:45:59.505735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:11.032382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10755,"ondisk_log_size":10755,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":371443,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":402,"num_write_kb":563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":742886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":367,"ondisk_log_size":367,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574904,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39628,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332212,"statfs":{"total":96636764160,"available":96596185088,"internally_reserved":0,"allocated":331776,"data_stored":675500,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":40237514},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.59199999999999997,"15min":0.57299999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":5.0620000000000003,"5min":5.0620000000000003,"15min":5.0620000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.68500000000000005,"15min":0.60099999999999998},"min":{"1min":0.372,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":5.1859999999999999,"5min":5.1859999999999999,"15min":5.1859999999999999},"last":0.46800000000000003}]},{"osd":1,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.36799999999999999,"5min":0.314,"15min":0.314},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.61699999999999999,"15min":0.58999999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":3.6070000000000002,"5min":3.6070000000000002,"15min":3.6070000000000002},"last":0.60499999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":2.4929999999999999,"5min":2.4929999999999999,"15min":2.4929999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.63400000000000001,"15min":0.61599999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":0.51200000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.56899999999999995,"15min":0.58199999999999996},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.8490000000000002,"5min":3.8490000000000002,"15min":3.8490000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64800000000000002,"15min":0.60499999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.3160000000000001,"5min":4.0129999999999999,"15min":4.0129999999999999},"last":0.57399999999999995}]},{"osd":4,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.434,"15min":0.442},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.78000000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.37},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45000000000000001,"15min":0.49199999999999999},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67300000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.45100000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46300000000000002,"15min":0.48699999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.499,"15min":0.52200000000000002},"min":{"1min":0.32600000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.753,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.40699999999999997}]},{"osd":6,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.53300000000000003,"15min":0.56299999999999994},"min":{"1min":0.36399999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81100000000000005,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.44},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.504,"15min":0.54000000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72799999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.497}]}]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35812,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336028,"statfs":{"total":96636764160,"available":96600092672,"internally_reserved":0,"allocated":618496,"data_stored":448530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36043209},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":0.69499999999999995,"15min":0.625},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":8.468,"5min":8.468,"15min":8.468},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.67300000000000004,"15min":0.625},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":7.5739999999999998,"5min":7.5739999999999998,"15min":7.5739999999999998},"last":0.76900000000000002}]},{"osd":1,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.29299999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60799999999999998,"15min":0.63900000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.67400000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.028,"5min":2.431,"15min":2.431},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.466,"5min":0.78500000000000003,"15min":0.64100000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.9139999999999997,"5min":6.9139999999999997,"15min":6.9139999999999997},"last":0.84999999999999998}]},{"osd":3,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":0.67200000000000004,"15min":0.60999999999999999},"min":{"1min":0.373,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.2400000000000002,"5min":6.2400000000000002,"15min":6.2400000000000002},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.70899999999999996,"15min":0.66200000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.4500000000000002,"5min":3.4500000000000002,"15min":3.4500000000000002},"last":0.86699999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.41299999999999998,"15min":0.45100000000000001},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.54300000000000004,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.439,"15min":0.48299999999999998},"min":{"1min":0.23699999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.441}]},{"osd":5,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.48799999999999999,"15min":0.52800000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.51100000000000001,"15min":0.56499999999999995},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.73599999999999999,"5min":0.752,"15min":0.752},"last":0.49099999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.48199999999999998,"15min":0.52000000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.71299999999999997,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.54000000000000004,"15min":0.58099999999999996},"min":{"1min":0.34000000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.53600000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574904,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39572,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332268,"statfs":{"total":96636764160,"available":96596242432,"internally_reserved":0,"allocated":274432,"data_stored":86302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.58299999999999996,"15min":0.56299999999999994},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.411,"5min":3.411,"15min":3.411},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.117,"5min":3.117,"15min":3.117},"last":0.57099999999999995}]},{"osd":1,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55500000000000005,"15min":0.56100000000000005},"min":{"1min":0.29899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.64300000000000002,"15min":0.58899999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.843,"5min":4.3860000000000001,"15min":4.3860000000000001},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.67300000000000004,"15min":0.64300000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":4.0819999999999999,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.64700000000000002,"15min":0.61399999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":4.3170000000000002,"5min":4.3170000000000002,"15min":4.3170000000000002},"last":0.64900000000000002}]},{"osd":3,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.90700000000000003,"15min":0.68700000000000006},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.4690000000000003,"5min":28.707000000000001,"15min":28.707000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.68700000000000006,"15min":0.625},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.2599999999999998,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":0.58499999999999996}]},{"osd":4,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.41899999999999998,"15min":0.45100000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.45500000000000002,"15min":0.48099999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.70399999999999996,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.39000000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.441,"15min":0.45700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.67000000000000004,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.46500000000000002,"15min":0.48499999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.40600000000000003}]},{"osd":7,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.49099999999999999,"15min":0.52000000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65600000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.501,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.42499999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574905,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39552,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332288,"statfs":{"total":96636764160,"available":96596262912,"internally_reserved":0,"allocated":253952,"data_stored":84621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":3.7160000000000002,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.61099999999999999,"15min":0.59399999999999997},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.448,"5min":2.9980000000000002,"15min":2.9980000000000002},"last":0.61199999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.69599999999999995,"15min":0.63},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.3700000000000001,"5min":4.3700000000000001,"15min":4.3700000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.63500000000000001,"15min":0.61499999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.9020000000000001,"5min":3.9020000000000001,"15min":3.9020000000000001},"last":0.56399999999999995}]},{"osd":2,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":2.085,"5min":2.613,"15min":2.613},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.194,"5min":0.72499999999999998,"15min":0.65800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":7.9809999999999999,"5min":7.9809999999999999,"15min":7.9809999999999999},"last":0.54000000000000004}]},{"osd":3,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.65800000000000003,"15min":0.63400000000000001},"min":{"1min":0.36199999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.68600000000000005,"15min":0.68899999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.8130000000000002,"5min":2.8130000000000002,"15min":2.8130000000000002},"last":0.65100000000000002}]},{"osd":5,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.42799999999999999,"15min":0.46100000000000002},"min":{"1min":0.25600000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.373},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.438,"15min":0.47299999999999998},"min":{"1min":0.254,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61399999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.41099999999999998}]},{"osd":6,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.442,"15min":0.47999999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.628,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.48599999999999999,"15min":0.53400000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.63600000000000001,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.434}]},{"osd":7,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.48099999999999998,"15min":0.50900000000000001},"min":{"1min":0.308,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.499},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.54400000000000004,"15min":0.60399999999999998},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70499999999999996,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.54700000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574904,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39916,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331924,"statfs":{"total":96636764160,"available":96595890176,"internally_reserved":0,"allocated":626688,"data_stored":448810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40237512},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.57899999999999996,"15min":0.49199999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.8449999999999998,"5min":4.8449999999999998,"15min":4.8449999999999998},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.5529999999999999,"5min":0.77800000000000002,"15min":0.59799999999999998},"min":{"1min":0.41099999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":7.7489999999999997,"5min":7.7489999999999997,"15min":7.7489999999999997},"last":0.42599999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":2.0529999999999999,"5min":0.84299999999999997,"15min":0.60499999999999998},"min":{"1min":0.317,"5min":0.252,"15min":0.252},"max":{"1min":28.908999999999999,"5min":28.908999999999999,"15min":28.908999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":0.73299999999999998,"15min":0.64600000000000002},"min":{"1min":0.40600000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.1520000000000001,"5min":3.1699999999999999,"15min":3.1699999999999999},"last":0.45300000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":0.80300000000000005,"15min":0.63500000000000001},"min":{"1min":0.41799999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":4.3630000000000004,"5min":4.3630000000000004,"15min":4.3630000000000004},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.76900000000000002,"15min":0.69299999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.1859999999999999,"5min":3.2610000000000001,"15min":3.2610000000000001},"last":0.60499999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":0.94299999999999995,"15min":0.82699999999999996},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":4.1029999999999998,"5min":4.1029999999999998,"15min":4.1029999999999998},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.79300000000000004,"15min":0.753},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.903,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":0.754}]},{"osd":5,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":0.80700000000000005,"15min":0.754},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":2.012,"5min":2.5790000000000002,"15min":2.5790000000000002},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.1160000000000001,"5min":0.89100000000000001,"15min":0.84699999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.0739999999999998,"5min":3.1970000000000001,"15min":3.1970000000000001},"last":0.71699999999999997}]},{"osd":6,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":0.96699999999999997,"15min":0.84199999999999997},"min":{"1min":0.61099999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":4.1449999999999996,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.0760000000000001,"5min":0.86499999999999999,"15min":0.82799999999999996},"min":{"1min":0.53000000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.9319999999999999,"5min":2.7360000000000002,"15min":2.7360000000000002},"last":0.79800000000000004}]},{"osd":7,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":0.88500000000000001,"15min":0.82399999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":3.117,"5min":3.2320000000000002,"15min":3.2320000000000002},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.613,"5min":1.006,"15min":0.874},"min":{"1min":0.56599999999999995,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":4.181,"5min":4.181,"15min":4.181},"last":0.86399999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35464,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336376,"statfs":{"total":96636764160,"available":96600449024,"internally_reserved":0,"allocated":262144,"data_stored":84385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.988,"5min":0.996,"15min":0.65800000000000003},"min":{"1min":0.26600000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":31.440999999999999,"5min":31.440999999999999,"15min":31.440999999999999},"last":0.437},{"interface":"front","average":{"1min":1.294,"5min":0.69099999999999995,"15min":0.57699999999999996},"min":{"1min":0.34899999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.7720000000000002,"5min":4.7720000000000002,"15min":4.7720000000000002},"last":0.47199999999999998}]},{"osd":1,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.2949999999999999,"5min":0.89800000000000002,"15min":0.65600000000000003},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.497},{"interface":"front","average":{"1min":1.121,"5min":0.755,"15min":0.61299999999999999},"min":{"1min":0.379,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.0230000000000001,"5min":6.5659999999999998,"15min":6.5659999999999998},"last":0.59099999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":0.71499999999999997,"15min":0.59299999999999997},"min":{"1min":0.371,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":6.5469999999999997,"5min":6.5469999999999997,"15min":6.5469999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.7869999999999999,"5min":0.84499999999999997,"15min":0.68300000000000005},"min":{"1min":0.45500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":8.8659999999999997,"5min":8.8659999999999997,"15min":8.8659999999999997},"last":0.57899999999999996}]},{"osd":4,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":0.81499999999999995,"15min":0.752},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":3.4460000000000002,"5min":3.4460000000000002,"15min":3.4460000000000002},"last":0.63},{"interface":"front","average":{"1min":1.163,"5min":0.80600000000000005,"15min":0.76300000000000001},"min":{"1min":0.46200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":6.4379999999999997,"5min":6.4379999999999997,"15min":6.4379999999999997},"last":0.76800000000000002}]},{"osd":5,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.929,"5min":0.97599999999999998,"15min":0.81699999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":9.1039999999999992,"5min":9.1039999999999992,"15min":9.1039999999999992},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.1899999999999999,"5min":0.84399999999999997,"15min":0.77400000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":3.633,"5min":3.633,"15min":3.633},"last":0.75}]},{"osd":6,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":0.83199999999999996,"15min":0.79400000000000004},"min":{"1min":0.54400000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.95,"5min":0.999,"15min":0.85899999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":9.141,"5min":9.141,"15min":9.141},"last":0.79200000000000004}]},{"osd":7,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":0.88400000000000001,"15min":0.84299999999999997},"min":{"1min":0.50700000000000001,"5min":0.505,"15min":0.505},"max":{"1min":3.5990000000000002,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.2450000000000001,"5min":0.86699999999999999,"15min":0.83499999999999996},"min":{"1min":0.57599999999999996,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":6.4800000000000004,"5min":6.4800000000000004,"15min":6.4800000000000004},"last":0.84199999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35460,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336380,"statfs":{"total":96636764160,"available":96600453120,"internally_reserved":0,"allocated":258048,"data_stored":84564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5630000000000002,"5min":1.0009999999999999,"15min":0.65200000000000002},"min":{"1min":0.29399999999999998,"5min":0.249,"15min":0.249},"max":{"1min":22.721,"5min":22.721,"15min":22.721},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.226,"5min":0.86699999999999999,"15min":0.64100000000000001},"min":{"1min":0.32200000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.2430000000000003,"5min":19.494,"15min":19.494},"last":0.69699999999999995}]},{"osd":2,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3570000000000002,"5min":0.98999999999999999,"15min":0.67200000000000004},"min":{"1min":0.34799999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":15.491,"5min":15.491,"15min":15.491},"last":0.57999999999999996},{"interface":"front","average":{"1min":1.5549999999999999,"5min":0.93300000000000005,"15min":0.70399999999999996},"min":{"1min":0.39600000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.6829999999999998,"5min":17.071000000000002,"15min":17.071000000000002},"last":0.50800000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.631,"5min":0.98599999999999999,"15min":0.73499999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":6.883,"5min":18.838000000000001,"15min":18.838000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.339,"5min":0.92200000000000004,"15min":0.69699999999999995},"min":{"1min":0.40899999999999997,"5min":0.31,"15min":0.31},"max":{"1min":7.0339999999999998,"5min":20.199000000000002,"15min":20.199000000000002},"last":0.63}]},{"osd":4,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.778,"5min":1.2390000000000001,"15min":0.91300000000000003},"min":{"1min":0.39800000000000002,"5min":0.373,"15min":0.373},"max":{"1min":15.654999999999999,"5min":15.654999999999999,"15min":15.654999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.091,"15min":0.89500000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":7.5659999999999998,"5min":18.742000000000001,"15min":18.742000000000001},"last":0.82099999999999995}]},{"osd":5,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":0.97699999999999998,"15min":0.89300000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":7.3230000000000004,"5min":7.3230000000000004,"15min":7.3230000000000004},"last":0.84599999999999997},{"interface":"front","average":{"1min":2.819,"5min":1.296,"15min":0.98299999999999998},"min":{"1min":0.51400000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":15.686,"5min":15.686,"15min":15.686},"last":0.87}]},{"osd":6,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":0.96799999999999997,"15min":0.86599999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":7.3700000000000001,"5min":7.3700000000000001,"15min":7.3700000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":2.851,"5min":1.3180000000000001,"15min":1.002},"min":{"1min":0.55500000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":15.718,"5min":15.718,"15min":15.718},"last":0.83799999999999997}]},{"osd":7,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":1.119,"15min":0.89500000000000002},"min":{"1min":0.495,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":8.1539999999999999,"5min":18.798999999999999,"15min":18.798999999999999},"last":0.88},{"interface":"front","average":{"1min":1.4339999999999999,"5min":0.995,"15min":0.90200000000000002},"min":{"1min":0.53000000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":7.407,"5min":7.4950000000000001,"15min":7.4950000000000001},"last":0.76000000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574904,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39616,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332224,"statfs":{"total":96636764160,"available":96596197376,"internally_reserved":0,"allocated":319488,"data_stored":673303,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.65800000000000003,"15min":0.53000000000000003},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.9239999999999999,"5min":7.2539999999999996,"15min":7.2539999999999996},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.2589999999999999,"5min":0.71999999999999997,"15min":0.55700000000000005},"min":{"1min":0.32400000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":6.0949999999999998,"5min":6.0949999999999998,"15min":6.0949999999999998},"last":0.68899999999999995}]},{"osd":2,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.78300000000000003,"15min":0.627},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.1800000000000002,"5min":7.1920000000000002,"15min":7.1920000000000002},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.151,"5min":0.74299999999999999,"15min":0.59899999999999998},"min":{"1min":0.36399999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":6.5,"5min":6.5,"15min":6.5},"last":0.51200000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":0.66500000000000004,"15min":0.54800000000000004},"min":{"1min":0.38800000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":4.4459999999999997,"5min":4.4459999999999997,"15min":4.4459999999999997},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.85999999999999999,"15min":0.69099999999999995},"min":{"1min":0.38700000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0409999999999999,"5min":10.522,"15min":10.522},"last":0.59799999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":0.89300000000000002,"15min":0.80100000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":6.4560000000000004,"5min":6.4560000000000004,"15min":6.4560000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.82799999999999996,"15min":0.77900000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":3.79,"5min":3.79,"15min":3.79},"last":0.76000000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":0.85399999999999998,"15min":0.78800000000000003},"min":{"1min":0.51300000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.8340000000000001,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":1.002,"5min":0.96599999999999997,"15min":0.85399999999999998},"min":{"1min":0.48599999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":4.5620000000000003,"5min":7.8339999999999996,"15min":7.8339999999999996},"last":0.70799999999999996}]},{"osd":6,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.98199999999999998,"15min":0.86499999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.7270000000000003,"5min":7.5510000000000002,"15min":7.5510000000000002},"last":0.754},{"interface":"front","average":{"1min":1.345,"5min":0.95699999999999996,"15min":0.81499999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":7.444,"5min":7.444,"15min":7.444},"last":0.78000000000000003}]},{"osd":7,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":0.95399999999999996,"15min":0.81599999999999995},"min":{"1min":0.51900000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":7.5350000000000001,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.1639999999999999,"5min":0.874,"15min":0.79700000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.1879999999999997,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":365730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":389120,"data_stored":365450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:48:23.603 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T16:48:23.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:48:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.793+0000 7fb34b200640 1 Processor -- start 2024-09-13T16:48:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.794+0000 7fb34b200640 1 -- start start 2024-09-13T16:48:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.795+0000 7fb34b200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb34414f790 0x7fb3440cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.795+0000 7fb34b200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb3440c42f0 0x7fb344070640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.795+0000 7fb34b200640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb34406cb60 con 0x7fb34414fb70 2024-09-13T16:48:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.795+0000 7fb34b200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3440c1c80 con 0x7fb3440c42f0 2024-09-13T16:48:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.795+0000 7fb34b200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34406ce40 con 0x7fb34414f790 2024-09-13T16:48:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.795+0000 7fb349e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb34414f790 0x7fb3440cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.795+0000 7fb34a800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb3440c42f0 0x7fb344070640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.795+0000 7fb34a800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb3440c42f0 0x7fb344070640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:38684/0 (socket says 172.21.15.28:38684) 2024-09-13T16:48:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.795+0000 7fb34a800640 1 -- 172.21.15.28:0/3690832256 learned_addr learned my addr 172.21.15.28:0/3690832256 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.796+0000 7fb349e00640 1 -- 172.21.15.28:0/3690832256 >> v2:172.21.15.191:3300/0 conn(0x7fb3440c42f0 msgr2=0x7fb344070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.796+0000 7fb349e00640 1 --2- 172.21.15.28:0/3690832256 >> v2:172.21.15.191:3300/0 conn(0x7fb3440c42f0 0x7fb344070640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.796+0000 7fb349e00640 1 -- 172.21.15.28:0/3690832256 >> v1:172.21.15.191:6789/0 conn(0x7fb34414fb70 legacy=0x7fb3440c3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.796+0000 7fb349e00640 1 -- 172.21.15.28:0/3690832256 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb344156070 con 0x7fb34414f790 2024-09-13T16:48:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.797+0000 7fb349e00640 1 --2- 172.21.15.28:0/3690832256 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb34414f790 0x7fb3440cabd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb3380097a0 tx=0x7fb33802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87657ec31336c3cb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.797+0000 7fb348a00640 1 -- 172.21.15.28:0/3690832256 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338032440 con 0x7fb34414f790 2024-09-13T16:48:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.797+0000 7fb348a00640 1 -- 172.21.15.28:0/3690832256 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3380325a0 con 0x7fb34414f790 2024-09-13T16:48:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.797+0000 7fb348a00640 1 -- 172.21.15.28:0/3690832256 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338032930 con 0x7fb34414f790 2024-09-13T16:48:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.798+0000 7fb34b200640 1 -- 172.21.15.28:0/3690832256 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb34414f790 msgr2=0x7fb3440cabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.798+0000 7fb34b200640 1 --2- 172.21.15.28:0/3690832256 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb34414f790 0x7fb3440cabd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb3380097a0 tx=0x7fb33802a210 comp rx=0 tx=0).stop 2024-09-13T16:48:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.798+0000 7fb34b200640 1 -- 172.21.15.28:0/3690832256 shutdown_connections 2024-09-13T16:48:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.798+0000 7fb34b200640 1 --2- 172.21.15.28:0/3690832256 >> v2:172.21.15.191:3300/0 conn(0x7fb3440c42f0 0x7fb344070640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.798+0000 7fb34b200640 1 --2- 172.21.15.28:0/3690832256 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb34414f790 0x7fb3440cabd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.798+0000 7fb34b200640 1 -- 172.21.15.28:0/3690832256 >> 172.21.15.28:0/3690832256 conn(0x7fb3440ca0e0 msgr2=0x7fb3440ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.798+0000 7fb34b200640 1 -- 172.21.15.28:0/3690832256 shutdown_connections 2024-09-13T16:48:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.798+0000 7fb34b200640 1 -- 172.21.15.28:0/3690832256 wait complete. 2024-09-13T16:48:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.799+0000 7fb34b200640 1 Processor -- start 2024-09-13T16:48:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.799+0000 7fb34b200640 1 -- start start 2024-09-13T16:48:23.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.800+0000 7fb34b200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3440c42f0 0x7fb3441496f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.800+0000 7fb34b200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb34414f790 0x7fb344142980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.800+0000 7fb34b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb344149c30 0x7fb344142ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.800+0000 7fb34b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3441ec060 con 0x7fb344149c30 2024-09-13T16:48:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.800+0000 7fb34b200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3441ec2c0 con 0x7fb3440c42f0 2024-09-13T16:48:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.800+0000 7fb34b200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34414ff60 con 0x7fb34414f790 2024-09-13T16:48:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.801+0000 7fb34a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb344149c30 0x7fb344142ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.801+0000 7fb349e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3440c42f0 0x7fb3441496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.801+0000 7fb34a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb344149c30 0x7fb344142ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39164/0 (socket says 172.21.15.28:39164) 2024-09-13T16:48:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.801+0000 7fb34a800640 1 -- 172.21.15.28:0/1446253157 learned_addr learned my addr 172.21.15.28:0/1446253157 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.801+0000 7fb349400640 1 --2- 172.21.15.28:0/1446253157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb34414f790 0x7fb344142980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.801+0000 7fb34a800640 1 -- 172.21.15.28:0/1446253157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb34414f790 msgr2=0x7fb344142980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.801+0000 7fb34a800640 1 --2- 172.21.15.28:0/1446253157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb34414f790 0x7fb344142980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.801+0000 7fb34a800640 1 -- 172.21.15.28:0/1446253157 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3440c42f0 msgr2=0x7fb3441496f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.801+0000 7fb34a800640 1 --2- 172.21.15.28:0/1446253157 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3440c42f0 0x7fb3441496f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.801+0000 7fb34a800640 1 -- 172.21.15.28:0/1446253157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb338009400 con 0x7fb344149c30 2024-09-13T16:48:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.801+0000 7fb34a800640 1 --2- 172.21.15.28:0/1446253157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb344149c30 0x7fb344142ec0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb34000efc0 tx=0x7fb34000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.801+0000 7fb33e000640 1 -- 172.21.15.28:0/1446253157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34000cd20 con 0x7fb344149c30 2024-09-13T16:48:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.802+0000 7fb33e000640 1 -- 172.21.15.28:0/1446253157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34000f040 con 0x7fb344149c30 2024-09-13T16:48:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.802+0000 7fb33e000640 1 -- 172.21.15.28:0/1446253157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb340003bd0 con 0x7fb344149c30 2024-09-13T16:48:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.802+0000 7fb34b200640 1 -- 172.21.15.28:0/1446253157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb344150240 con 0x7fb344149c30 2024-09-13T16:48:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.802+0000 7fb34b200640 1 -- 172.21.15.28:0/1446253157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb344150800 con 0x7fb344149c30 2024-09-13T16:48:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.803+0000 7fb34b200640 1 -- 172.21.15.28:0/1446253157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34406fcd0 con 0x7fb344149c30 2024-09-13T16:48:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.804+0000 7fb33e000640 1 -- 172.21.15.28:0/1446253157 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb340003d30 con 0x7fb344149c30 2024-09-13T16:48:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.805+0000 7fb33e000640 1 --2- 172.21.15.28:0/1446253157 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb32806ac30 0x7fb32806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.805+0000 7fb33e000640 1 -- 172.21.15.28:0/1446253157 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb3400899e0 con 0x7fb344149c30 2024-09-13T16:48:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.805+0000 7fb349e00640 1 --2- 172.21.15.28:0/1446253157 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb32806ac30 0x7fb32806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.805+0000 7fb349e00640 1 --2- 172.21.15.28:0/1446253157 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb32806ac30 0x7fb32806d0f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb338034000 tx=0x7fb33802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.809+0000 7fb33e000640 1 -- 172.21.15.28:0/1446253157 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb340054ac0 con 0x7fb344149c30 2024-09-13T16:48:23.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.990+0000 7fb34b200640 1 -- 172.21.15.28:0/1446253157 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb3440c1760 con 0x7fb32806ac30 2024-09-13T16:48:23.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:23.998+0000 7fb33e000640 1 -- 172.21.15.28:0/1446253157 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289778 (secure 0 0 0) 0x7fb338040070 con 0x7fb32806ac30 2024-09-13T16:48:24.003 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:48:24.005 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T16:48:24.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.002+0000 7fb34b200640 1 -- 172.21.15.28:0/1446253157 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb32806ac30 msgr2=0x7fb32806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:24.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.002+0000 7fb34b200640 1 --2- 172.21.15.28:0/1446253157 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb32806ac30 0x7fb32806d0f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb338034000 tx=0x7fb33802e000 comp rx=0 tx=0).stop 2024-09-13T16:48:24.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.002+0000 7fb34b200640 1 -- 172.21.15.28:0/1446253157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb344149c30 msgr2=0x7fb344142ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:24.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.002+0000 7fb34b200640 1 --2- 172.21.15.28:0/1446253157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb344149c30 0x7fb344142ec0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb34000efc0 tx=0x7fb34000c490 comp rx=0 tx=0).stop 2024-09-13T16:48:24.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.002+0000 7fb34b200640 1 -- 172.21.15.28:0/1446253157 shutdown_connections 2024-09-13T16:48:24.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.002+0000 7fb34b200640 1 --2- 172.21.15.28:0/1446253157 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fb32806ac30 0x7fb32806d0f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:24.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.002+0000 7fb34b200640 1 --2- 172.21.15.28:0/1446253157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb344149c30 0x7fb344142ec0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:24.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.002+0000 7fb34b200640 1 --2- 172.21.15.28:0/1446253157 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb34414f790 0x7fb344142980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:24.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.002+0000 7fb34b200640 1 --2- 172.21.15.28:0/1446253157 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb3440c42f0 0x7fb3441496f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:24.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.002+0000 7fb34b200640 1 -- 172.21.15.28:0/1446253157 >> 172.21.15.28:0/1446253157 conn(0x7fb3440ca0e0 msgr2=0x7fb3440ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:24.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.002+0000 7fb34b200640 1 -- 172.21.15.28:0/1446253157 shutdown_connections 2024-09-13T16:48:24.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.002+0000 7fb34b200640 1 -- 172.21.15.28:0/1446253157 wait complete. 2024-09-13T16:48:24.019 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-09-13T16:48:23.653032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":961811,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":121,"num_read_kb":105,"num_write":9777,"num_write_kb":25168803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11239,"ondisk_log_size":11239,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":305020,"kb_used_data":2876,"kb_used_omap":12,"kb_used_meta":302067,"kb_avail":754669700,"statfs":{"total":773094113280,"available":772781772800,"internally_reserved":0,"allocated":2945024,"data_stored":2586015,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12743,"internal_metadata":309317177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":700,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002932"},"pg_stats":[{"pgid":"3.3f","version":"22'179","reported_seq":193,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.519062+0000","last_change":"2024-09-13T16:43:54.987241+0000","last_active":"2024-09-13T16:47:59.519062+0000","last_peered":"2024-09-13T16:47:59.519062+0000","last_clean":"2024-09-13T16:47:59.519062+0000","last_became_active":"2024-09-13T16:43:54.986988+0000","last_became_peered":"2024-09-13T16:43:54.986988+0000","last_unstale":"2024-09-13T16:47:59.519062+0000","last_undegraded":"2024-09-13T16:47:59.519062+0000","last_fullsized":"2024-09-13T16:47:59.519062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"22'134","reported_seq":148,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.454727+0000","last_change":"2024-09-13T16:43:55.001141+0000","last_active":"2024-09-13T16:47:59.454727+0000","last_peered":"2024-09-13T16:47:59.454727+0000","last_clean":"2024-09-13T16:47:59.454727+0000","last_became_active":"2024-09-13T16:43:55.000757+0000","last_became_peered":"2024-09-13T16:43:55.000757+0000","last_unstale":"2024-09-13T16:47:59.454727+0000","last_undegraded":"2024-09-13T16:47:59.454727+0000","last_fullsized":"2024-09-13T16:47:59.454727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"22'205","reported_seq":219,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.510237+0000","last_change":"2024-09-13T16:43:54.986925+0000","last_active":"2024-09-13T16:47:59.510237+0000","last_peered":"2024-09-13T16:47:59.510237+0000","last_clean":"2024-09-13T16:47:59.510237+0000","last_became_active":"2024-09-13T16:43:54.986198+0000","last_became_peered":"2024-09-13T16:43:54.986198+0000","last_unstale":"2024-09-13T16:47:59.510237+0000","last_undegraded":"2024-09-13T16:47:59.510237+0000","last_fullsized":"2024-09-13T16:47:59.510237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:03.127550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"22'125","reported_seq":139,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.433692+0000","last_change":"2024-09-13T16:43:54.992249+0000","last_active":"2024-09-13T16:47:59.433692+0000","last_peered":"2024-09-13T16:47:59.433692+0000","last_clean":"2024-09-13T16:47:59.433692+0000","last_became_active":"2024-09-13T16:43:54.991804+0000","last_became_peered":"2024-09-13T16:43:54.991804+0000","last_unstale":"2024-09-13T16:47:59.433692+0000","last_undegraded":"2024-09-13T16:47:59.433692+0000","last_fullsized":"2024-09-13T16:47:59.433692+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":270336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"22'190","reported_seq":203,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.430620+0000","last_change":"2024-09-13T16:43:54.987564+0000","last_active":"2024-09-13T16:47:59.430620+0000","last_peered":"2024-09-13T16:47:59.430620+0000","last_clean":"2024-09-13T16:47:59.430620+0000","last_became_active":"2024-09-13T16:43:54.987328+0000","last_became_peered":"2024-09-13T16:43:54.987328+0000","last_unstale":"2024-09-13T16:47:59.430620+0000","last_undegraded":"2024-09-13T16:47:59.430620+0000","last_fullsized":"2024-09-13T16:47:59.430620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:36.006730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"22'150","reported_seq":163,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.494433+0000","last_change":"2024-09-13T16:43:55.000019+0000","last_active":"2024-09-13T16:47:59.494433+0000","last_peered":"2024-09-13T16:47:59.494433+0000","last_clean":"2024-09-13T16:47:59.494433+0000","last_became_active":"2024-09-13T16:43:54.999752+0000","last_became_peered":"2024-09-13T16:43:54.999752+0000","last_unstale":"2024-09-13T16:47:59.494433+0000","last_undegraded":"2024-09-13T16:47:59.494433+0000","last_fullsized":"2024-09-13T16:47:59.494433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"22'170","reported_seq":183,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.493499+0000","last_change":"2024-09-13T16:43:54.999533+0000","last_active":"2024-09-13T16:47:59.493499+0000","last_peered":"2024-09-13T16:47:59.493499+0000","last_clean":"2024-09-13T16:47:59.493499+0000","last_became_active":"2024-09-13T16:43:54.999350+0000","last_became_peered":"2024-09-13T16:43:54.999350+0000","last_unstale":"2024-09-13T16:47:59.493499+0000","last_undegraded":"2024-09-13T16:47:59.493499+0000","last_fullsized":"2024-09-13T16:47:59.493499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:29.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"22'156","reported_seq":169,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.511303+0000","last_change":"2024-09-13T16:43:54.998526+0000","last_active":"2024-09-13T16:47:59.511303+0000","last_peered":"2024-09-13T16:47:59.511303+0000","last_clean":"2024-09-13T16:47:59.511303+0000","last_became_active":"2024-09-13T16:43:54.996309+0000","last_became_peered":"2024-09-13T16:43:54.996309+0000","last_unstale":"2024-09-13T16:47:59.511303+0000","last_undegraded":"2024-09-13T16:47:59.511303+0000","last_fullsized":"2024-09-13T16:47:59.511303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"22'144","reported_seq":157,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.520032+0000","last_change":"2024-09-13T16:43:54.993786+0000","last_active":"2024-09-13T16:47:59.520032+0000","last_peered":"2024-09-13T16:47:59.520032+0000","last_clean":"2024-09-13T16:47:59.520032+0000","last_became_active":"2024-09-13T16:43:54.993475+0000","last_became_peered":"2024-09-13T16:43:54.993475+0000","last_unstale":"2024-09-13T16:47:59.520032+0000","last_undegraded":"2024-09-13T16:47:59.520032+0000","last_fullsized":"2024-09-13T16:47:59.520032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"22'174","reported_seq":187,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.513207+0000","last_change":"2024-09-13T16:43:54.986272+0000","last_active":"2024-09-13T16:47:59.513207+0000","last_peered":"2024-09-13T16:47:59.513207+0000","last_clean":"2024-09-13T16:47:59.513207+0000","last_became_active":"2024-09-13T16:43:54.985979+0000","last_became_peered":"2024-09-13T16:43:54.985979+0000","last_unstale":"2024-09-13T16:47:59.513207+0000","last_undegraded":"2024-09-13T16:47:59.513207+0000","last_fullsized":"2024-09-13T16:47:59.513207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"22'168","reported_seq":181,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.448009+0000","last_change":"2024-09-13T16:43:54.986207+0000","last_active":"2024-09-13T16:47:59.448009+0000","last_peered":"2024-09-13T16:47:59.448009+0000","last_clean":"2024-09-13T16:47:59.448009+0000","last_became_active":"2024-09-13T16:43:54.985948+0000","last_became_peered":"2024-09-13T16:43:54.985948+0000","last_unstale":"2024-09-13T16:47:59.448009+0000","last_undegraded":"2024-09-13T16:47:59.448009+0000","last_fullsized":"2024-09-13T16:47:59.448009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"22'176","reported_seq":189,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.438078+0000","last_change":"2024-09-13T16:43:54.986636+0000","last_active":"2024-09-13T16:47:59.438078+0000","last_peered":"2024-09-13T16:47:59.438078+0000","last_clean":"2024-09-13T16:47:59.438078+0000","last_became_active":"2024-09-13T16:43:54.986161+0000","last_became_peered":"2024-09-13T16:43:54.986161+0000","last_unstale":"2024-09-13T16:47:59.438078+0000","last_undegraded":"2024-09-13T16:47:59.438078+0000","last_fullsized":"2024-09-13T16:47:59.438078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:09.340330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"22'160","reported_seq":173,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.511135+0000","last_change":"2024-09-13T16:43:54.997783+0000","last_active":"2024-09-13T16:47:59.511135+0000","last_peered":"2024-09-13T16:47:59.511135+0000","last_clean":"2024-09-13T16:47:59.511135+0000","last_became_active":"2024-09-13T16:43:54.997162+0000","last_became_peered":"2024-09-13T16:43:54.997162+0000","last_unstale":"2024-09-13T16:47:59.511135+0000","last_undegraded":"2024-09-13T16:47:59.511135+0000","last_fullsized":"2024-09-13T16:47:59.511135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"22'173","reported_seq":186,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.497364+0000","last_change":"2024-09-13T16:43:54.995262+0000","last_active":"2024-09-13T16:47:59.497364+0000","last_peered":"2024-09-13T16:47:59.497364+0000","last_clean":"2024-09-13T16:47:59.497364+0000","last_became_active":"2024-09-13T16:43:54.994891+0000","last_became_peered":"2024-09-13T16:43:54.994891+0000","last_unstale":"2024-09-13T16:47:59.497364+0000","last_undegraded":"2024-09-13T16:47:59.497364+0000","last_fullsized":"2024-09-13T16:47:59.497364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"22'185","reported_seq":198,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.492832+0000","last_change":"2024-09-13T16:43:54.981534+0000","last_active":"2024-09-13T16:47:59.492832+0000","last_peered":"2024-09-13T16:47:59.492832+0000","last_clean":"2024-09-13T16:47:59.492832+0000","last_became_active":"2024-09-13T16:43:54.981191+0000","last_became_peered":"2024-09-13T16:43:54.981191+0000","last_unstale":"2024-09-13T16:47:59.492832+0000","last_undegraded":"2024-09-13T16:47:59.492832+0000","last_fullsized":"2024-09-13T16:47:59.492832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"22'151","reported_seq":164,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.506334+0000","last_change":"2024-09-13T16:43:54.994157+0000","last_active":"2024-09-13T16:47:59.506334+0000","last_peered":"2024-09-13T16:47:59.506334+0000","last_clean":"2024-09-13T16:47:59.506334+0000","last_became_active":"2024-09-13T16:43:54.993915+0000","last_became_peered":"2024-09-13T16:43:54.993915+0000","last_unstale":"2024-09-13T16:47:59.506334+0000","last_undegraded":"2024-09-13T16:47:59.506334+0000","last_fullsized":"2024-09-13T16:47:59.506334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"22'167","reported_seq":180,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.509298+0000","last_change":"2024-09-13T16:43:54.987328+0000","last_active":"2024-09-13T16:47:59.509298+0000","last_peered":"2024-09-13T16:47:59.509298+0000","last_clean":"2024-09-13T16:47:59.509298+0000","last_became_active":"2024-09-13T16:43:54.987048+0000","last_became_peered":"2024-09-13T16:43:54.987048+0000","last_unstale":"2024-09-13T16:47:59.509298+0000","last_undegraded":"2024-09-13T16:47:59.509298+0000","last_fullsized":"2024-09-13T16:47:59.509298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"22'132","reported_seq":145,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.442769+0000","last_change":"2024-09-13T16:43:54.985628+0000","last_active":"2024-09-13T16:47:59.442769+0000","last_peered":"2024-09-13T16:47:59.442769+0000","last_clean":"2024-09-13T16:47:59.442769+0000","last_became_active":"2024-09-13T16:43:54.985378+0000","last_became_peered":"2024-09-13T16:43:54.985378+0000","last_unstale":"2024-09-13T16:47:59.442769+0000","last_undegraded":"2024-09-13T16:47:59.442769+0000","last_fullsized":"2024-09-13T16:47:59.442769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"22'150","reported_seq":163,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.510652+0000","last_change":"2024-09-13T16:43:54.996712+0000","last_active":"2024-09-13T16:47:59.510652+0000","last_peered":"2024-09-13T16:47:59.510652+0000","last_clean":"2024-09-13T16:47:59.510652+0000","last_became_active":"2024-09-13T16:43:54.996523+0000","last_became_peered":"2024-09-13T16:43:54.996523+0000","last_unstale":"2024-09-13T16:47:59.510652+0000","last_undegraded":"2024-09-13T16:47:59.510652+0000","last_fullsized":"2024-09-13T16:47:59.510652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"22'159","reported_seq":172,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.501849+0000","last_change":"2024-09-13T16:43:54.984879+0000","last_active":"2024-09-13T16:47:59.501849+0000","last_peered":"2024-09-13T16:47:59.501849+0000","last_clean":"2024-09-13T16:47:59.501849+0000","last_became_active":"2024-09-13T16:43:54.984595+0000","last_became_peered":"2024-09-13T16:43:54.984595+0000","last_unstale":"2024-09-13T16:47:59.501849+0000","last_undegraded":"2024-09-13T16:47:59.501849+0000","last_fullsized":"2024-09-13T16:47:59.501849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"22'156","reported_seq":169,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.509526+0000","last_change":"2024-09-13T16:43:54.996457+0000","last_active":"2024-09-13T16:47:59.509526+0000","last_peered":"2024-09-13T16:47:59.509526+0000","last_clean":"2024-09-13T16:47:59.509526+0000","last_became_active":"2024-09-13T16:43:54.996275+0000","last_became_peered":"2024-09-13T16:43:54.996275+0000","last_unstale":"2024-09-13T16:47:59.509526+0000","last_undegraded":"2024-09-13T16:47:59.509526+0000","last_fullsized":"2024-09-13T16:47:59.509526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"22'189","reported_seq":202,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.497242+0000","last_change":"2024-09-13T16:43:54.985368+0000","last_active":"2024-09-13T16:47:59.497242+0000","last_peered":"2024-09-13T16:47:59.497242+0000","last_clean":"2024-09-13T16:47:59.497242+0000","last_became_active":"2024-09-13T16:43:54.985130+0000","last_became_peered":"2024-09-13T16:43:54.985130+0000","last_unstale":"2024-09-13T16:47:59.497242+0000","last_undegraded":"2024-09-13T16:47:59.497242+0000","last_fullsized":"2024-09-13T16:47:59.497242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"22'150","reported_seq":163,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.439460+0000","last_change":"2024-09-13T16:43:54.991862+0000","last_active":"2024-09-13T16:47:59.439460+0000","last_peered":"2024-09-13T16:47:59.439460+0000","last_clean":"2024-09-13T16:47:59.439460+0000","last_became_active":"2024-09-13T16:43:54.991554+0000","last_became_peered":"2024-09-13T16:43:54.991554+0000","last_unstale":"2024-09-13T16:47:59.439460+0000","last_undegraded":"2024-09-13T16:47:59.439460+0000","last_fullsized":"2024-09-13T16:47:59.439460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"22'156","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.506350+0000","last_change":"2024-09-13T16:44:59.221091+0000","last_active":"2024-09-13T16:47:59.506350+0000","last_peered":"2024-09-13T16:47:59.506350+0000","last_clean":"2024-09-13T16:47:59.506350+0000","last_became_active":"2024-09-13T16:44:59.220762+0000","last_became_peered":"2024-09-13T16:44:59.220762+0000","last_unstale":"2024-09-13T16:47:59.506350+0000","last_undegraded":"2024-09-13T16:47:59.506350+0000","last_fullsized":"2024-09-13T16:47:59.506350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:39.907088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"22'168","reported_seq":182,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.440760+0000","last_change":"2024-09-13T16:43:54.991366+0000","last_active":"2024-09-13T16:47:59.440760+0000","last_peered":"2024-09-13T16:47:59.440760+0000","last_clean":"2024-09-13T16:47:59.440760+0000","last_became_active":"2024-09-13T16:43:54.991127+0000","last_became_peered":"2024-09-13T16:43:54.991127+0000","last_unstale":"2024-09-13T16:47:59.440760+0000","last_undegraded":"2024-09-13T16:47:59.440760+0000","last_fullsized":"2024-09-13T16:47:59.440760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"22'165","reported_seq":179,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.519235+0000","last_change":"2024-09-13T16:43:54.985615+0000","last_active":"2024-09-13T16:47:59.519235+0000","last_peered":"2024-09-13T16:47:59.519235+0000","last_clean":"2024-09-13T16:47:59.519235+0000","last_became_active":"2024-09-13T16:43:54.985299+0000","last_became_peered":"2024-09-13T16:43:54.985299+0000","last_unstale":"2024-09-13T16:47:59.519235+0000","last_undegraded":"2024-09-13T16:47:59.519235+0000","last_fullsized":"2024-09-13T16:47:59.519235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"22'141","reported_seq":155,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.515984+0000","last_change":"2024-09-13T16:43:54.993325+0000","last_active":"2024-09-13T16:47:59.515984+0000","last_peered":"2024-09-13T16:47:59.515984+0000","last_clean":"2024-09-13T16:47:59.515984+0000","last_became_active":"2024-09-13T16:43:54.993009+0000","last_became_peered":"2024-09-13T16:43:54.993009+0000","last_unstale":"2024-09-13T16:47:59.515984+0000","last_undegraded":"2024-09-13T16:47:59.515984+0000","last_fullsized":"2024-09-13T16:47:59.515984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"22'184","reported_seq":198,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.508028+0000","last_change":"2024-09-13T16:43:54.998229+0000","last_active":"2024-09-13T16:47:59.508028+0000","last_peered":"2024-09-13T16:47:59.508028+0000","last_clean":"2024-09-13T16:47:59.508028+0000","last_became_active":"2024-09-13T16:43:54.995774+0000","last_became_peered":"2024-09-13T16:43:54.995774+0000","last_unstale":"2024-09-13T16:47:59.508028+0000","last_undegraded":"2024-09-13T16:47:59.508028+0000","last_fullsized":"2024-09-13T16:47:59.508028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"22'165","reported_seq":179,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.514352+0000","last_change":"2024-09-13T16:43:55.002126+0000","last_active":"2024-09-13T16:47:59.514352+0000","last_peered":"2024-09-13T16:47:59.514352+0000","last_clean":"2024-09-13T16:47:59.514352+0000","last_became_active":"2024-09-13T16:43:55.001713+0000","last_became_peered":"2024-09-13T16:43:55.001713+0000","last_unstale":"2024-09-13T16:47:59.514352+0000","last_undegraded":"2024-09-13T16:47:59.514352+0000","last_fullsized":"2024-09-13T16:47:59.514352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"22'148","reported_seq":162,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.517055+0000","last_change":"2024-09-13T16:43:54.992745+0000","last_active":"2024-09-13T16:47:59.517055+0000","last_peered":"2024-09-13T16:47:59.517055+0000","last_clean":"2024-09-13T16:47:59.517055+0000","last_became_active":"2024-09-13T16:43:54.992508+0000","last_became_peered":"2024-09-13T16:43:54.992508+0000","last_unstale":"2024-09-13T16:47:59.517055+0000","last_undegraded":"2024-09-13T16:47:59.517055+0000","last_fullsized":"2024-09-13T16:47:59.517055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:23.913470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"22'131","reported_seq":145,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.520706+0000","last_change":"2024-09-13T16:43:54.990631+0000","last_active":"2024-09-13T16:47:59.520706+0000","last_peered":"2024-09-13T16:47:59.520706+0000","last_clean":"2024-09-13T16:47:59.520706+0000","last_became_active":"2024-09-13T16:43:54.986277+0000","last_became_peered":"2024-09-13T16:43:54.986277+0000","last_unstale":"2024-09-13T16:47:59.520706+0000","last_undegraded":"2024-09-13T16:47:59.520706+0000","last_fullsized":"2024-09-13T16:47:59.520706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"22'175","reported_seq":189,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.499494+0000","last_change":"2024-09-13T16:43:55.000059+0000","last_active":"2024-09-13T16:47:59.499494+0000","last_peered":"2024-09-13T16:47:59.499494+0000","last_clean":"2024-09-13T16:47:59.499494+0000","last_became_active":"2024-09-13T16:43:54.999778+0000","last_became_peered":"2024-09-13T16:43:54.999778+0000","last_unstale":"2024-09-13T16:47:59.499494+0000","last_undegraded":"2024-09-13T16:47:59.499494+0000","last_fullsized":"2024-09-13T16:47:59.499494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"22'212","reported_seq":225,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.503806+0000","last_change":"2024-09-13T16:43:54.988375+0000","last_active":"2024-09-13T16:47:59.503806+0000","last_peered":"2024-09-13T16:47:59.503806+0000","last_clean":"2024-09-13T16:47:59.503806+0000","last_became_active":"2024-09-13T16:43:54.988131+0000","last_became_peered":"2024-09-13T16:43:54.988131+0000","last_unstale":"2024-09-13T16:47:59.503806+0000","last_undegraded":"2024-09-13T16:47:59.503806+0000","last_fullsized":"2024-09-13T16:47:59.503806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:36.635755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.219925+0000","last_change":"2024-09-13T16:43:54.555042+0000","last_active":"2024-09-13T16:44:59.219925+0000","last_peered":"2024-09-13T16:44:59.219925+0000","last_clean":"2024-09-13T16:44:59.219925+0000","last_became_active":"2024-09-13T16:43:54.554697+0000","last_became_peered":"2024-09-13T16:43:54.554697+0000","last_unstale":"2024-09-13T16:44:59.219925+0000","last_undegraded":"2024-09-13T16:44:59.219925+0000","last_fullsized":"2024-09-13T16:44:59.219925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"22'165","reported_seq":178,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.517461+0000","last_change":"2024-09-13T16:43:54.998755+0000","last_active":"2024-09-13T16:47:59.517461+0000","last_peered":"2024-09-13T16:47:59.517461+0000","last_clean":"2024-09-13T16:47:59.517461+0000","last_became_active":"2024-09-13T16:43:54.998399+0000","last_became_peered":"2024-09-13T16:43:54.998399+0000","last_unstale":"2024-09-13T16:47:59.517461+0000","last_undegraded":"2024-09-13T16:47:59.517461+0000","last_fullsized":"2024-09-13T16:47:59.517461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.215854+0000","last_change":"2024-09-13T16:43:54.553250+0000","last_active":"2024-09-13T16:44:59.215854+0000","last_peered":"2024-09-13T16:44:59.215854+0000","last_clean":"2024-09-13T16:44:59.215854+0000","last_became_active":"2024-09-13T16:43:54.552977+0000","last_became_peered":"2024-09-13T16:43:54.552977+0000","last_unstale":"2024-09-13T16:44:59.215854+0000","last_undegraded":"2024-09-13T16:44:59.215854+0000","last_fullsized":"2024-09-13T16:44:59.215854+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.219531+0000","last_change":"2024-09-13T16:43:54.558065+0000","last_active":"2024-09-13T16:44:59.219531+0000","last_peered":"2024-09-13T16:44:59.219531+0000","last_clean":"2024-09-13T16:44:59.219531+0000","last_became_active":"2024-09-13T16:43:54.557722+0000","last_became_peered":"2024-09-13T16:43:54.557722+0000","last_unstale":"2024-09-13T16:44:59.219531+0000","last_undegraded":"2024-09-13T16:44:59.219531+0000","last_fullsized":"2024-09-13T16:44:59.219531+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"22'151","reported_seq":164,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.495858+0000","last_change":"2024-09-13T16:43:55.002041+0000","last_active":"2024-09-13T16:47:59.495858+0000","last_peered":"2024-09-13T16:47:59.495858+0000","last_clean":"2024-09-13T16:47:59.495858+0000","last_became_active":"2024-09-13T16:43:55.001512+0000","last_became_peered":"2024-09-13T16:43:55.001512+0000","last_unstale":"2024-09-13T16:47:59.495858+0000","last_undegraded":"2024-09-13T16:47:59.495858+0000","last_fullsized":"2024-09-13T16:47:59.495858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.383328+0000","last_change":"2024-09-13T16:43:54.550389+0000","last_active":"2024-09-13T16:44:59.383328+0000","last_peered":"2024-09-13T16:44:59.383328+0000","last_clean":"2024-09-13T16:44:59.383328+0000","last_became_active":"2024-09-13T16:43:54.550154+0000","last_became_peered":"2024-09-13T16:43:54.550154+0000","last_unstale":"2024-09-13T16:44:59.383328+0000","last_undegraded":"2024-09-13T16:44:59.383328+0000","last_fullsized":"2024-09-13T16:44:59.383328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"22'168","reported_seq":181,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.449760+0000","last_change":"2024-09-13T16:43:54.993377+0000","last_active":"2024-09-13T16:47:59.449760+0000","last_peered":"2024-09-13T16:47:59.449760+0000","last_clean":"2024-09-13T16:47:59.449760+0000","last_became_active":"2024-09-13T16:43:54.993036+0000","last_became_peered":"2024-09-13T16:43:54.993036+0000","last_unstale":"2024-09-13T16:47:59.449760+0000","last_undegraded":"2024-09-13T16:47:59.449760+0000","last_fullsized":"2024-09-13T16:47:59.449760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:13.068438+0000","last_change":"2024-09-13T16:43:54.555528+0000","last_active":"2024-09-13T16:46:13.068438+0000","last_peered":"2024-09-13T16:46:13.068438+0000","last_clean":"2024-09-13T16:46:13.068438+0000","last_became_active":"2024-09-13T16:43:54.555267+0000","last_became_peered":"2024-09-13T16:43:54.555267+0000","last_unstale":"2024-09-13T16:46:13.068438+0000","last_undegraded":"2024-09-13T16:46:13.068438+0000","last_fullsized":"2024-09-13T16:46:13.068438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"22'143","reported_seq":156,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.438299+0000","last_change":"2024-09-13T16:43:54.981706+0000","last_active":"2024-09-13T16:47:59.438299+0000","last_peered":"2024-09-13T16:47:59.438299+0000","last_clean":"2024-09-13T16:47:59.438299+0000","last_became_active":"2024-09-13T16:43:54.981429+0000","last_became_peered":"2024-09-13T16:43:54.981429+0000","last_unstale":"2024-09-13T16:47:59.438299+0000","last_undegraded":"2024-09-13T16:47:59.438299+0000","last_fullsized":"2024-09-13T16:47:59.438299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.382875+0000","last_change":"2024-09-13T16:43:54.554593+0000","last_active":"2024-09-13T16:44:59.382875+0000","last_peered":"2024-09-13T16:44:59.382875+0000","last_clean":"2024-09-13T16:44:59.382875+0000","last_became_active":"2024-09-13T16:43:54.554307+0000","last_became_peered":"2024-09-13T16:43:54.554307+0000","last_unstale":"2024-09-13T16:44:59.382875+0000","last_undegraded":"2024-09-13T16:44:59.382875+0000","last_fullsized":"2024-09-13T16:44:59.382875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"22'195","reported_seq":208,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.497022+0000","last_change":"2024-09-13T16:43:54.993485+0000","last_active":"2024-09-13T16:47:59.497022+0000","last_peered":"2024-09-13T16:47:59.497022+0000","last_clean":"2024-09-13T16:47:59.497022+0000","last_became_active":"2024-09-13T16:43:54.993226+0000","last_became_peered":"2024-09-13T16:43:54.993226+0000","last_unstale":"2024-09-13T16:47:59.497022+0000","last_undegraded":"2024-09-13T16:47:59.497022+0000","last_fullsized":"2024-09-13T16:47:59.497022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'11","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:03.123692+0000","last_change":"2024-09-13T16:43:54.562976+0000","last_active":"2024-09-13T16:48:03.123692+0000","last_peered":"2024-09-13T16:48:03.123692+0000","last_clean":"2024-09-13T16:48:03.123692+0000","last_became_active":"2024-09-13T16:43:54.560480+0000","last_became_peered":"2024-09-13T16:43:54.560480+0000","last_unstale":"2024-09-13T16:48:03.123692+0000","last_undegraded":"2024-09-13T16:48:03.123692+0000","last_fullsized":"2024-09-13T16:48:03.123692+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"22'200","reported_seq":213,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.501250+0000","last_change":"2024-09-13T16:43:54.986459+0000","last_active":"2024-09-13T16:47:59.501250+0000","last_peered":"2024-09-13T16:47:59.501250+0000","last_clean":"2024-09-13T16:47:59.501250+0000","last_became_active":"2024-09-13T16:43:54.986083+0000","last_became_peered":"2024-09-13T16:43:54.986083+0000","last_unstale":"2024-09-13T16:47:59.501250+0000","last_undegraded":"2024-09-13T16:47:59.501250+0000","last_fullsized":"2024-09-13T16:47:59.501250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:51.960459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'165","reported_seq":179,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:14.014679+0000","last_change":"2024-09-13T16:43:54.565929+0000","last_active":"2024-09-13T16:48:14.014679+0000","last_peered":"2024-09-13T16:48:14.014679+0000","last_clean":"2024-09-13T16:48:14.014679+0000","last_became_active":"2024-09-13T16:43:54.565701+0000","last_became_peered":"2024-09-13T16:43:54.565701+0000","last_unstale":"2024-09-13T16:48:14.014679+0000","last_undegraded":"2024-09-13T16:48:14.014679+0000","last_fullsized":"2024-09-13T16:48:14.014679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364449,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":166,"num_write_kb":479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"22'235","reported_seq":248,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.515163+0000","last_change":"2024-09-13T16:43:54.989068+0000","last_active":"2024-09-13T16:47:59.515163+0000","last_peered":"2024-09-13T16:47:59.515163+0000","last_clean":"2024-09-13T16:47:59.515163+0000","last_became_active":"2024-09-13T16:43:54.988779+0000","last_became_peered":"2024-09-13T16:43:54.988779+0000","last_unstale":"2024-09-13T16:47:59.515163+0000","last_undegraded":"2024-09-13T16:47:59.515163+0000","last_fullsized":"2024-09-13T16:47:59.515163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":213,"num_write_kb":581632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.138534+0000","last_change":"2024-09-13T16:43:54.561487+0000","last_active":"2024-09-13T16:44:59.138534+0000","last_peered":"2024-09-13T16:44:59.138534+0000","last_clean":"2024-09-13T16:44:59.138534+0000","last_became_active":"2024-09-13T16:43:54.561077+0000","last_became_peered":"2024-09-13T16:43:54.561077+0000","last_unstale":"2024-09-13T16:44:59.138534+0000","last_undegraded":"2024-09-13T16:44:59.138534+0000","last_fullsized":"2024-09-13T16:44:59.138534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"22'153","reported_seq":166,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.500527+0000","last_change":"2024-09-13T16:43:54.999517+0000","last_active":"2024-09-13T16:47:59.500527+0000","last_peered":"2024-09-13T16:47:59.500527+0000","last_clean":"2024-09-13T16:47:59.500527+0000","last_became_active":"2024-09-13T16:43:54.999283+0000","last_became_peered":"2024-09-13T16:43:54.999283+0000","last_unstale":"2024-09-13T16:47:59.500527+0000","last_undegraded":"2024-09-13T16:47:59.500527+0000","last_fullsized":"2024-09-13T16:47:59.500527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:20.207775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.383950+0000","last_change":"2024-09-13T16:43:54.542871+0000","last_active":"2024-09-13T16:44:59.383950+0000","last_peered":"2024-09-13T16:44:59.383950+0000","last_clean":"2024-09-13T16:44:59.383950+0000","last_became_active":"2024-09-13T16:43:54.542548+0000","last_became_peered":"2024-09-13T16:43:54.542548+0000","last_unstale":"2024-09-13T16:44:59.383950+0000","last_undegraded":"2024-09-13T16:44:59.383950+0000","last_fullsized":"2024-09-13T16:44:59.383950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"22'180","reported_seq":193,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.452606+0000","last_change":"2024-09-13T16:43:54.989168+0000","last_active":"2024-09-13T16:47:59.452606+0000","last_peered":"2024-09-13T16:47:59.452606+0000","last_clean":"2024-09-13T16:47:59.452606+0000","last_became_active":"2024-09-13T16:43:54.988716+0000","last_became_peered":"2024-09-13T16:43:54.988716+0000","last_unstale":"2024-09-13T16:47:59.452606+0000","last_undegraded":"2024-09-13T16:47:59.452606+0000","last_fullsized":"2024-09-13T16:47:59.452606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"20'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.215766+0000","last_change":"2024-09-13T16:43:54.551551+0000","last_active":"2024-09-13T16:44:59.215766+0000","last_peered":"2024-09-13T16:44:59.215766+0000","last_clean":"2024-09-13T16:44:59.215766+0000","last_became_active":"2024-09-13T16:43:54.551293+0000","last_became_peered":"2024-09-13T16:43:54.551293+0000","last_unstale":"2024-09-13T16:44:59.215766+0000","last_undegraded":"2024-09-13T16:44:59.215766+0000","last_fullsized":"2024-09-13T16:44:59.215766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":799,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"22'163","reported_seq":176,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.499067+0000","last_change":"2024-09-13T16:43:54.994516+0000","last_active":"2024-09-13T16:47:59.499067+0000","last_peered":"2024-09-13T16:47:59.499067+0000","last_clean":"2024-09-13T16:47:59.499067+0000","last_became_active":"2024-09-13T16:43:54.994245+0000","last_became_peered":"2024-09-13T16:43:54.994245+0000","last_unstale":"2024-09-13T16:47:59.499067+0000","last_undegraded":"2024-09-13T16:47:59.499067+0000","last_fullsized":"2024-09-13T16:47:59.499067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:34.277032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'18","reported_seq":32,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:14.507552+0000","last_change":"2024-09-13T16:43:54.562232+0000","last_active":"2024-09-13T16:46:14.507552+0000","last_peered":"2024-09-13T16:46:14.507552+0000","last_clean":"2024-09-13T16:46:14.507552+0000","last_became_active":"2024-09-13T16:43:54.561939+0000","last_became_peered":"2024-09-13T16:43:54.561939+0000","last_unstale":"2024-09-13T16:46:14.507552+0000","last_undegraded":"2024-09-13T16:46:14.507552+0000","last_fullsized":"2024-09-13T16:46:14.507552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"22'169","reported_seq":182,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.452403+0000","last_change":"2024-09-13T16:43:54.988439+0000","last_active":"2024-09-13T16:47:59.452403+0000","last_peered":"2024-09-13T16:47:59.452403+0000","last_clean":"2024-09-13T16:47:59.452403+0000","last_became_active":"2024-09-13T16:43:54.988172+0000","last_became_peered":"2024-09-13T16:43:54.988172+0000","last_unstale":"2024-09-13T16:47:59.452403+0000","last_undegraded":"2024-09-13T16:47:59.452403+0000","last_fullsized":"2024-09-13T16:47:59.452403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:58.954977+0000","last_change":"2024-09-13T16:43:54.559372+0000","last_active":"2024-09-13T16:44:58.954977+0000","last_peered":"2024-09-13T16:44:58.954977+0000","last_clean":"2024-09-13T16:44:58.954977+0000","last_became_active":"2024-09-13T16:43:54.559111+0000","last_became_peered":"2024-09-13T16:43:54.559111+0000","last_unstale":"2024-09-13T16:44:58.954977+0000","last_undegraded":"2024-09-13T16:44:58.954977+0000","last_fullsized":"2024-09-13T16:44:58.954977+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:40.399883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'151","reported_seq":164,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.509322+0000","last_change":"2024-09-13T16:43:54.988313+0000","last_active":"2024-09-13T16:47:59.509322+0000","last_peered":"2024-09-13T16:47:59.509322+0000","last_clean":"2024-09-13T16:47:59.509322+0000","last_became_active":"2024-09-13T16:43:54.988059+0000","last_became_peered":"2024-09-13T16:43:54.988059+0000","last_unstale":"2024-09-13T16:47:59.509322+0000","last_undegraded":"2024-09-13T16:47:59.509322+0000","last_fullsized":"2024-09-13T16:47:59.509322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:58.955244+0000","last_change":"2024-09-13T16:43:54.566410+0000","last_active":"2024-09-13T16:44:58.955244+0000","last_peered":"2024-09-13T16:44:58.955244+0000","last_clean":"2024-09-13T16:44:58.955244+0000","last_became_active":"2024-09-13T16:43:54.565861+0000","last_became_peered":"2024-09-13T16:43:54.565861+0000","last_unstale":"2024-09-13T16:44:58.955244+0000","last_undegraded":"2024-09-13T16:44:58.955244+0000","last_fullsized":"2024-09-13T16:44:58.955244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:11.794353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"22'127","reported_seq":138,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.428730+0000","last_change":"2024-09-13T16:44:58.956863+0000","last_active":"2024-09-13T16:47:59.428730+0000","last_peered":"2024-09-13T16:47:59.428730+0000","last_clean":"2024-09-13T16:47:59.428730+0000","last_became_active":"2024-09-13T16:44:58.956557+0000","last_became_peered":"2024-09-13T16:44:58.956557+0000","last_unstale":"2024-09-13T16:47:59.428730+0000","last_undegraded":"2024-09-13T16:47:59.428730+0000","last_fullsized":"2024-09-13T16:47:59.428730+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:10.217411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":278528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:58.957395+0000","last_change":"2024-09-13T16:43:54.559723+0000","last_active":"2024-09-13T16:44:58.957395+0000","last_peered":"2024-09-13T16:44:58.957395+0000","last_clean":"2024-09-13T16:44:58.957395+0000","last_became_active":"2024-09-13T16:43:54.559353+0000","last_became_peered":"2024-09-13T16:43:54.559353+0000","last_unstale":"2024-09-13T16:44:58.957395+0000","last_undegraded":"2024-09-13T16:44:58.957395+0000","last_fullsized":"2024-09-13T16:44:58.957395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"22'203","reported_seq":216,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.498887+0000","last_change":"2024-09-13T16:43:54.992511+0000","last_active":"2024-09-13T16:47:59.498887+0000","last_peered":"2024-09-13T16:47:59.498887+0000","last_clean":"2024-09-13T16:47:59.498887+0000","last_became_active":"2024-09-13T16:43:54.992249+0000","last_became_peered":"2024-09-13T16:43:54.992249+0000","last_unstale":"2024-09-13T16:47:59.498887+0000","last_undegraded":"2024-09-13T16:47:59.498887+0000","last_fullsized":"2024-09-13T16:47:59.498887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"20'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:58.955067+0000","last_change":"2024-09-13T16:43:54.551390+0000","last_active":"2024-09-13T16:44:58.955067+0000","last_peered":"2024-09-13T16:44:58.955067+0000","last_clean":"2024-09-13T16:44:58.955067+0000","last_became_active":"2024-09-13T16:43:54.547074+0000","last_became_peered":"2024-09-13T16:43:54.547074+0000","last_unstale":"2024-09-13T16:44:58.955067+0000","last_undegraded":"2024-09-13T16:44:58.955067+0000","last_fullsized":"2024-09-13T16:44:58.955067+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"22'165","reported_seq":178,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.492614+0000","last_change":"2024-09-13T16:43:54.990458+0000","last_active":"2024-09-13T16:47:59.492614+0000","last_peered":"2024-09-13T16:47:59.492614+0000","last_clean":"2024-09-13T16:47:59.492614+0000","last_became_active":"2024-09-13T16:43:54.990196+0000","last_became_peered":"2024-09-13T16:43:54.990196+0000","last_unstale":"2024-09-13T16:47:59.492614+0000","last_undegraded":"2024-09-13T16:47:59.492614+0000","last_fullsized":"2024-09-13T16:47:59.492614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:54.505478+0000","last_change":"2024-09-13T16:43:51.914639+0000","last_active":"2024-09-13T16:45:54.505478+0000","last_peered":"2024-09-13T16:45:54.505478+0000","last_clean":"2024-09-13T16:45:54.505478+0000","last_became_active":"2024-09-13T16:43:51.913797+0000","last_became_peered":"2024-09-13T16:43:51.913797+0000","last_unstale":"2024-09-13T16:45:54.505478+0000","last_undegraded":"2024-09-13T16:45:54.505478+0000","last_fullsized":"2024-09-13T16:45:54.505478+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_clean_scrub_stamp":"2024-09-13T16:43:50.498116+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:12.730911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:00.501470+0000","last_change":"2024-09-13T16:43:54.564153+0000","last_active":"2024-09-13T16:46:00.501470+0000","last_peered":"2024-09-13T16:46:00.501470+0000","last_clean":"2024-09-13T16:46:00.501470+0000","last_became_active":"2024-09-13T16:43:54.563949+0000","last_became_peered":"2024-09-13T16:43:54.563949+0000","last_unstale":"2024-09-13T16:46:00.501470+0000","last_undegraded":"2024-09-13T16:46:00.501470+0000","last_fullsized":"2024-09-13T16:46:00.501470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"22'207","reported_seq":218,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.437738+0000","last_change":"2024-09-13T16:44:59.387015+0000","last_active":"2024-09-13T16:47:59.437738+0000","last_peered":"2024-09-13T16:47:59.437738+0000","last_clean":"2024-09-13T16:47:59.437738+0000","last_became_active":"2024-09-13T16:44:59.386756+0000","last_became_peered":"2024-09-13T16:44:59.386756+0000","last_unstale":"2024-09-13T16:47:59.437738+0000","last_undegraded":"2024-09-13T16:47:59.437738+0000","last_fullsized":"2024-09-13T16:47:59.437738+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:15.100276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":186,"num_write_kb":507904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:54.505444+0000","last_change":"2024-09-13T16:43:54.564804+0000","last_active":"2024-09-13T16:45:54.505444+0000","last_peered":"2024-09-13T16:45:54.505444+0000","last_clean":"2024-09-13T16:45:54.505444+0000","last_became_active":"2024-09-13T16:43:54.564605+0000","last_became_peered":"2024-09-13T16:43:54.564605+0000","last_unstale":"2024-09-13T16:45:54.505444+0000","last_undegraded":"2024-09-13T16:45:54.505444+0000","last_fullsized":"2024-09-13T16:45:54.505444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"22'187","reported_seq":201,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.514834+0000","last_change":"2024-09-13T16:43:54.997884+0000","last_active":"2024-09-13T16:47:59.514834+0000","last_peered":"2024-09-13T16:47:59.514834+0000","last_clean":"2024-09-13T16:47:59.514834+0000","last_became_active":"2024-09-13T16:43:54.997510+0000","last_became_peered":"2024-09-13T16:43:54.997510+0000","last_unstale":"2024-09-13T16:47:59.514834+0000","last_undegraded":"2024-09-13T16:47:59.514834+0000","last_fullsized":"2024-09-13T16:47:59.514834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:15.251424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'12","reported_seq":32,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:10.123144+0000","last_change":"2024-09-13T16:43:54.566294+0000","last_active":"2024-09-13T16:48:10.123144+0000","last_peered":"2024-09-13T16:48:10.123144+0000","last_clean":"2024-09-13T16:48:10.123144+0000","last_became_active":"2024-09-13T16:43:54.566023+0000","last_became_peered":"2024-09-13T16:43:54.566023+0000","last_unstale":"2024-09-13T16:48:10.123144+0000","last_undegraded":"2024-09-13T16:48:10.123144+0000","last_fullsized":"2024-09-13T16:48:10.123144+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"22'159","reported_seq":173,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.518351+0000","last_change":"2024-09-13T16:43:54.981116+0000","last_active":"2024-09-13T16:47:59.518351+0000","last_peered":"2024-09-13T16:47:59.518351+0000","last_clean":"2024-09-13T16:47:59.518351+0000","last_became_active":"2024-09-13T16:43:54.980766+0000","last_became_peered":"2024-09-13T16:43:54.980766+0000","last_unstale":"2024-09-13T16:47:59.518351+0000","last_undegraded":"2024-09-13T16:47:59.518351+0000","last_fullsized":"2024-09-13T16:47:59.518351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:00.501452+0000","last_change":"2024-09-13T16:43:54.563716+0000","last_active":"2024-09-13T16:46:00.501452+0000","last_peered":"2024-09-13T16:46:00.501452+0000","last_clean":"2024-09-13T16:46:00.501452+0000","last_became_active":"2024-09-13T16:43:54.563522+0000","last_became_peered":"2024-09-13T16:43:54.563522+0000","last_unstale":"2024-09-13T16:46:00.501452+0000","last_undegraded":"2024-09-13T16:46:00.501452+0000","last_fullsized":"2024-09-13T16:46:00.501452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"22'156","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.513433+0000","last_change":"2024-09-13T16:43:54.999051+0000","last_active":"2024-09-13T16:47:59.513433+0000","last_peered":"2024-09-13T16:47:59.513433+0000","last_clean":"2024-09-13T16:47:59.513433+0000","last_became_active":"2024-09-13T16:43:54.998803+0000","last_became_peered":"2024-09-13T16:43:54.998803+0000","last_unstale":"2024-09-13T16:47:59.513433+0000","last_undegraded":"2024-09-13T16:47:59.513433+0000","last_fullsized":"2024-09-13T16:47:59.513433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:13.687404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:59.505325+0000","last_change":"2024-09-13T16:43:54.569945+0000","last_active":"2024-09-13T16:45:59.505325+0000","last_peered":"2024-09-13T16:45:59.505325+0000","last_clean":"2024-09-13T16:45:59.505325+0000","last_became_active":"2024-09-13T16:43:54.568846+0000","last_became_peered":"2024-09-13T16:43:54.568846+0000","last_unstale":"2024-09-13T16:45:59.505325+0000","last_undegraded":"2024-09-13T16:45:59.505325+0000","last_fullsized":"2024-09-13T16:45:59.505325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"22'175","reported_seq":189,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.514055+0000","last_change":"2024-09-13T16:43:54.992082+0000","last_active":"2024-09-13T16:47:59.514055+0000","last_peered":"2024-09-13T16:47:59.514055+0000","last_clean":"2024-09-13T16:47:59.514055+0000","last_became_active":"2024-09-13T16:43:54.991659+0000","last_became_peered":"2024-09-13T16:43:54.991659+0000","last_unstale":"2024-09-13T16:47:59.514055+0000","last_undegraded":"2024-09-13T16:47:59.514055+0000","last_fullsized":"2024-09-13T16:47:59.514055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:59.506045+0000","last_change":"2024-09-13T16:43:54.569111+0000","last_active":"2024-09-13T16:45:59.506045+0000","last_peered":"2024-09-13T16:45:59.506045+0000","last_clean":"2024-09-13T16:45:59.506045+0000","last_became_active":"2024-09-13T16:43:54.568903+0000","last_became_peered":"2024-09-13T16:43:54.568903+0000","last_unstale":"2024-09-13T16:45:59.506045+0000","last_undegraded":"2024-09-13T16:45:59.506045+0000","last_fullsized":"2024-09-13T16:45:59.506045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"22'197","reported_seq":211,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.518940+0000","last_change":"2024-09-13T16:43:54.980100+0000","last_active":"2024-09-13T16:47:59.518940+0000","last_peered":"2024-09-13T16:47:59.518940+0000","last_clean":"2024-09-13T16:47:59.518940+0000","last_became_active":"2024-09-13T16:43:54.979584+0000","last_became_peered":"2024-09-13T16:43:54.979584+0000","last_unstale":"2024-09-13T16:47:59.518940+0000","last_undegraded":"2024-09-13T16:47:59.518940+0000","last_fullsized":"2024-09-13T16:47:59.518940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'20","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:14.014651+0000","last_change":"2024-09-13T16:43:54.560710+0000","last_active":"2024-09-13T16:48:14.014651+0000","last_peered":"2024-09-13T16:48:14.014651+0000","last_clean":"2024-09-13T16:48:14.014651+0000","last_became_active":"2024-09-13T16:43:54.560030+0000","last_became_peered":"2024-09-13T16:43:54.560030+0000","last_unstale":"2024-09-13T16:48:14.014651+0000","last_undegraded":"2024-09-13T16:48:14.014651+0000","last_fullsized":"2024-09-13T16:48:14.014651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'191","reported_seq":205,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.510771+0000","last_change":"2024-09-13T16:43:54.990812+0000","last_active":"2024-09-13T16:47:59.510771+0000","last_peered":"2024-09-13T16:47:59.510771+0000","last_clean":"2024-09-13T16:47:59.510771+0000","last_became_active":"2024-09-13T16:43:54.990440+0000","last_became_peered":"2024-09-13T16:43:54.990440+0000","last_unstale":"2024-09-13T16:47:59.510771+0000","last_undegraded":"2024-09-13T16:47:59.510771+0000","last_fullsized":"2024-09-13T16:47:59.510771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:09.142916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'8","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:03.122078+0000","last_change":"2024-09-13T16:43:54.558439+0000","last_active":"2024-09-13T16:48:03.122078+0000","last_peered":"2024-09-13T16:48:03.122078+0000","last_clean":"2024-09-13T16:48:03.122078+0000","last_became_active":"2024-09-13T16:43:54.558194+0000","last_became_peered":"2024-09-13T16:43:54.558194+0000","last_unstale":"2024-09-13T16:48:03.122078+0000","last_undegraded":"2024-09-13T16:48:03.122078+0000","last_fullsized":"2024-09-13T16:48:03.122078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"22'154","reported_seq":168,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.507374+0000","last_change":"2024-09-13T16:43:54.995534+0000","last_active":"2024-09-13T16:47:59.507374+0000","last_peered":"2024-09-13T16:47:59.507374+0000","last_clean":"2024-09-13T16:47:59.507374+0000","last_became_active":"2024-09-13T16:43:54.995397+0000","last_became_peered":"2024-09-13T16:43:54.995397+0000","last_unstale":"2024-09-13T16:47:59.507374+0000","last_undegraded":"2024-09-13T16:47:59.507374+0000","last_fullsized":"2024-09-13T16:47:59.507374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:19.697247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:04.506432+0000","last_change":"2024-09-13T16:43:54.569085+0000","last_active":"2024-09-13T16:46:04.506432+0000","last_peered":"2024-09-13T16:46:04.506432+0000","last_clean":"2024-09-13T16:46:04.506432+0000","last_became_active":"2024-09-13T16:43:54.568892+0000","last_became_peered":"2024-09-13T16:43:54.568892+0000","last_unstale":"2024-09-13T16:46:04.506432+0000","last_undegraded":"2024-09-13T16:46:04.506432+0000","last_fullsized":"2024-09-13T16:46:04.506432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"22'196","reported_seq":210,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.498566+0000","last_change":"2024-09-13T16:43:54.984120+0000","last_active":"2024-09-13T16:47:59.498566+0000","last_peered":"2024-09-13T16:47:59.498566+0000","last_clean":"2024-09-13T16:47:59.498566+0000","last_became_active":"2024-09-13T16:43:54.983886+0000","last_became_peered":"2024-09-13T16:43:54.983886+0000","last_unstale":"2024-09-13T16:47:59.498566+0000","last_undegraded":"2024-09-13T16:47:59.498566+0000","last_fullsized":"2024-09-13T16:47:59.498566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:08.285605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:04.505849+0000","last_change":"2024-09-13T16:43:54.555092+0000","last_active":"2024-09-13T16:46:04.505849+0000","last_peered":"2024-09-13T16:46:04.505849+0000","last_clean":"2024-09-13T16:46:04.505849+0000","last_became_active":"2024-09-13T16:43:54.554230+0000","last_became_peered":"2024-09-13T16:43:54.554230+0000","last_unstale":"2024-09-13T16:46:04.505849+0000","last_undegraded":"2024-09-13T16:46:04.505849+0000","last_fullsized":"2024-09-13T16:46:04.505849+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"22'169","reported_seq":183,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.513275+0000","last_change":"2024-09-13T16:43:54.984588+0000","last_active":"2024-09-13T16:47:59.513275+0000","last_peered":"2024-09-13T16:47:59.513275+0000","last_clean":"2024-09-13T16:47:59.513275+0000","last_became_active":"2024-09-13T16:43:54.984221+0000","last_became_peered":"2024-09-13T16:43:54.984221+0000","last_unstale":"2024-09-13T16:47:59.513275+0000","last_undegraded":"2024-09-13T16:47:59.513275+0000","last_fullsized":"2024-09-13T16:47:59.513275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:04.505864+0000","last_change":"2024-09-13T16:43:54.562646+0000","last_active":"2024-09-13T16:46:04.505864+0000","last_peered":"2024-09-13T16:46:04.505864+0000","last_clean":"2024-09-13T16:46:04.505864+0000","last_became_active":"2024-09-13T16:43:54.556888+0000","last_became_peered":"2024-09-13T16:43:54.556888+0000","last_unstale":"2024-09-13T16:46:04.505864+0000","last_undegraded":"2024-09-13T16:46:04.505864+0000","last_fullsized":"2024-09-13T16:46:04.505864+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"22'209","reported_seq":222,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.429450+0000","last_change":"2024-09-13T16:43:54.999328+0000","last_active":"2024-09-13T16:47:59.429450+0000","last_peered":"2024-09-13T16:47:59.429450+0000","last_clean":"2024-09-13T16:47:59.429450+0000","last_became_active":"2024-09-13T16:43:54.999101+0000","last_became_peered":"2024-09-13T16:43:54.999101+0000","last_unstale":"2024-09-13T16:47:59.429450+0000","last_undegraded":"2024-09-13T16:47:59.429450+0000","last_fullsized":"2024-09-13T16:47:59.429450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:09.507033+0000","last_change":"2024-09-13T16:43:54.559214+0000","last_active":"2024-09-13T16:46:09.507033+0000","last_peered":"2024-09-13T16:46:09.507033+0000","last_clean":"2024-09-13T16:46:09.507033+0000","last_became_active":"2024-09-13T16:43:54.558233+0000","last_became_peered":"2024-09-13T16:43:54.558233+0000","last_unstale":"2024-09-13T16:46:09.507033+0000","last_undegraded":"2024-09-13T16:46:09.507033+0000","last_fullsized":"2024-09-13T16:46:09.507033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"22'185","reported_seq":199,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.519284+0000","last_change":"2024-09-13T16:43:54.986565+0000","last_active":"2024-09-13T16:47:59.519284+0000","last_peered":"2024-09-13T16:47:59.519284+0000","last_clean":"2024-09-13T16:47:59.519284+0000","last_became_active":"2024-09-13T16:43:54.986084+0000","last_became_peered":"2024-09-13T16:43:54.986084+0000","last_unstale":"2024-09-13T16:47:59.519284+0000","last_undegraded":"2024-09-13T16:47:59.519284+0000","last_fullsized":"2024-09-13T16:47:59.519284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:12.523122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"22'150","reported_seq":163,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.519001+0000","last_change":"2024-09-13T16:43:54.982611+0000","last_active":"2024-09-13T16:47:59.519001+0000","last_peered":"2024-09-13T16:47:59.519001+0000","last_clean":"2024-09-13T16:47:59.519001+0000","last_became_active":"2024-09-13T16:43:54.982338+0000","last_became_peered":"2024-09-13T16:43:54.982338+0000","last_unstale":"2024-09-13T16:47:59.519001+0000","last_undegraded":"2024-09-13T16:47:59.519001+0000","last_fullsized":"2024-09-13T16:47:59.519001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.382688+0000","last_change":"2024-09-13T16:43:54.558799+0000","last_active":"2024-09-13T16:44:59.382688+0000","last_peered":"2024-09-13T16:44:59.382688+0000","last_clean":"2024-09-13T16:44:59.382688+0000","last_became_active":"2024-09-13T16:43:54.558332+0000","last_became_peered":"2024-09-13T16:43:54.558332+0000","last_unstale":"2024-09-13T16:44:59.382688+0000","last_undegraded":"2024-09-13T16:44:59.382688+0000","last_fullsized":"2024-09-13T16:44:59.382688+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:19.201506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'143","reported_seq":154,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.415246+0000","last_change":"2024-09-13T16:44:58.957875+0000","last_active":"2024-09-13T16:47:59.415246+0000","last_peered":"2024-09-13T16:47:59.415246+0000","last_clean":"2024-09-13T16:47:59.415246+0000","last_became_active":"2024-09-13T16:44:58.957568+0000","last_became_peered":"2024-09-13T16:44:58.957568+0000","last_unstale":"2024-09-13T16:47:59.415246+0000","last_undegraded":"2024-09-13T16:47:59.415246+0000","last_fullsized":"2024-09-13T16:47:59.415246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:28.185704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'13","reported_seq":27,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:13.069441+0000","last_change":"2024-09-13T16:43:54.559551+0000","last_active":"2024-09-13T16:46:13.069441+0000","last_peered":"2024-09-13T16:46:13.069441+0000","last_clean":"2024-09-13T16:46:13.069441+0000","last_became_active":"2024-09-13T16:43:54.559290+0000","last_became_peered":"2024-09-13T16:43:54.559290+0000","last_unstale":"2024-09-13T16:46:13.069441+0000","last_undegraded":"2024-09-13T16:46:13.069441+0000","last_fullsized":"2024-09-13T16:46:13.069441+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"22'174","reported_seq":188,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.508073+0000","last_change":"2024-09-13T16:43:55.000783+0000","last_active":"2024-09-13T16:47:59.508073+0000","last_peered":"2024-09-13T16:47:59.508073+0000","last_clean":"2024-09-13T16:47:59.508073+0000","last_became_active":"2024-09-13T16:43:55.000196+0000","last_became_peered":"2024-09-13T16:43:55.000196+0000","last_unstale":"2024-09-13T16:47:59.508073+0000","last_undegraded":"2024-09-13T16:47:59.508073+0000","last_fullsized":"2024-09-13T16:47:59.508073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:02.640458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:54.504808+0000","last_change":"2024-09-13T16:43:54.559886+0000","last_active":"2024-09-13T16:45:54.504808+0000","last_peered":"2024-09-13T16:45:54.504808+0000","last_clean":"2024-09-13T16:45:54.504808+0000","last_became_active":"2024-09-13T16:43:54.559586+0000","last_became_peered":"2024-09-13T16:43:54.559586+0000","last_unstale":"2024-09-13T16:45:54.504808+0000","last_undegraded":"2024-09-13T16:45:54.504808+0000","last_fullsized":"2024-09-13T16:45:54.504808+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:13.876327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"22'147","reported_seq":161,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:59.520197+0000","last_change":"2024-09-13T16:43:54.996099+0000","last_active":"2024-09-13T16:47:59.520197+0000","last_peered":"2024-09-13T16:47:59.520197+0000","last_clean":"2024-09-13T16:47:59.520197+0000","last_became_active":"2024-09-13T16:43:54.995802+0000","last_became_peered":"2024-09-13T16:43:54.995802+0000","last_unstale":"2024-09-13T16:47:59.520197+0000","last_undegraded":"2024-09-13T16:47:59.520197+0000","last_fullsized":"2024-09-13T16:47:59.520197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":28,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:59.505735+0000","last_change":"2024-09-13T16:43:54.562699+0000","last_active":"2024-09-13T16:45:59.505735+0000","last_peered":"2024-09-13T16:45:59.505735+0000","last_clean":"2024-09-13T16:45:59.505735+0000","last_became_active":"2024-09-13T16:43:54.560552+0000","last_became_peered":"2024-09-13T16:43:54.560552+0000","last_unstale":"2024-09-13T16:45:59.505735+0000","last_undegraded":"2024-09-13T16:45:59.505735+0000","last_fullsized":"2024-09-13T16:45:59.505735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:11.032382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10755,"ondisk_log_size":10755,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":371443,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":402,"num_write_kb":563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":742886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":367,"ondisk_log_size":367,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574904,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39628,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332212,"statfs":{"total":96636764160,"available":96596185088,"internally_reserved":0,"allocated":331776,"data_stored":675500,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":40237514},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.59199999999999997,"15min":0.57299999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":5.0620000000000003,"5min":5.0620000000000003,"15min":5.0620000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.68500000000000005,"15min":0.60099999999999998},"min":{"1min":0.372,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":5.1859999999999999,"5min":5.1859999999999999,"15min":5.1859999999999999},"last":0.46800000000000003}]},{"osd":1,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.36799999999999999,"5min":0.314,"15min":0.314},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.61699999999999999,"15min":0.58999999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":3.6070000000000002,"5min":3.6070000000000002,"15min":3.6070000000000002},"last":0.60499999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":2.4929999999999999,"5min":2.4929999999999999,"15min":2.4929999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.63400000000000001,"15min":0.61599999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":0.51200000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.56899999999999995,"15min":0.58199999999999996},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.8490000000000002,"5min":3.8490000000000002,"15min":3.8490000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64800000000000002,"15min":0.60499999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.3160000000000001,"5min":4.0129999999999999,"15min":4.0129999999999999},"last":0.57399999999999995}]},{"osd":4,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.434,"15min":0.442},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.78000000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.37},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45000000000000001,"15min":0.49199999999999999},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67300000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.45100000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46300000000000002,"15min":0.48699999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.499,"15min":0.52200000000000002},"min":{"1min":0.32600000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.753,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.40699999999999997}]},{"osd":6,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.53300000000000003,"15min":0.56299999999999994},"min":{"1min":0.36399999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81100000000000005,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.44},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.504,"15min":0.54000000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72799999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.497}]}]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35812,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336028,"statfs":{"total":96636764160,"available":96600092672,"internally_reserved":0,"allocated":618496,"data_stored":448530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36043209},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":0.69499999999999995,"15min":0.625},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":8.468,"5min":8.468,"15min":8.468},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.67300000000000004,"15min":0.625},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":7.5739999999999998,"5min":7.5739999999999998,"15min":7.5739999999999998},"last":0.76900000000000002}]},{"osd":1,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.29299999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60799999999999998,"15min":0.63900000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.67400000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.028,"5min":2.431,"15min":2.431},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.466,"5min":0.78500000000000003,"15min":0.64100000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.9139999999999997,"5min":6.9139999999999997,"15min":6.9139999999999997},"last":0.84999999999999998}]},{"osd":3,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":0.67200000000000004,"15min":0.60999999999999999},"min":{"1min":0.373,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.2400000000000002,"5min":6.2400000000000002,"15min":6.2400000000000002},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.70899999999999996,"15min":0.66200000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.4500000000000002,"5min":3.4500000000000002,"15min":3.4500000000000002},"last":0.86699999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.41299999999999998,"15min":0.45100000000000001},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.54300000000000004,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.439,"15min":0.48299999999999998},"min":{"1min":0.23699999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.441}]},{"osd":5,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.48799999999999999,"15min":0.52800000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.51100000000000001,"15min":0.56499999999999995},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.73599999999999999,"5min":0.752,"15min":0.752},"last":0.49099999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.48199999999999998,"15min":0.52000000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.71299999999999997,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.54000000000000004,"15min":0.58099999999999996},"min":{"1min":0.34000000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.53600000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574904,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39572,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332268,"statfs":{"total":96636764160,"available":96596242432,"internally_reserved":0,"allocated":274432,"data_stored":86302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.58299999999999996,"15min":0.56299999999999994},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.411,"5min":3.411,"15min":3.411},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.117,"5min":3.117,"15min":3.117},"last":0.57099999999999995}]},{"osd":1,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55500000000000005,"15min":0.56100000000000005},"min":{"1min":0.29899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.64300000000000002,"15min":0.58899999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.843,"5min":4.3860000000000001,"15min":4.3860000000000001},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.67300000000000004,"15min":0.64300000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":4.0819999999999999,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.64700000000000002,"15min":0.61399999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":4.3170000000000002,"5min":4.3170000000000002,"15min":4.3170000000000002},"last":0.64900000000000002}]},{"osd":3,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.90700000000000003,"15min":0.68700000000000006},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.4690000000000003,"5min":28.707000000000001,"15min":28.707000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.68700000000000006,"15min":0.625},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.2599999999999998,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":0.58499999999999996}]},{"osd":4,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.41899999999999998,"15min":0.45100000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.45500000000000002,"15min":0.48099999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.70399999999999996,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.39000000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.441,"15min":0.45700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.67000000000000004,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.46500000000000002,"15min":0.48499999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.40600000000000003}]},{"osd":7,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.49099999999999999,"15min":0.52000000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65600000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.501,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.42499999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574905,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39552,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332288,"statfs":{"total":96636764160,"available":96596262912,"internally_reserved":0,"allocated":253952,"data_stored":84621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":3.7160000000000002,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.61099999999999999,"15min":0.59399999999999997},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.448,"5min":2.9980000000000002,"15min":2.9980000000000002},"last":0.61199999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.69599999999999995,"15min":0.63},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.3700000000000001,"5min":4.3700000000000001,"15min":4.3700000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.63500000000000001,"15min":0.61499999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.9020000000000001,"5min":3.9020000000000001,"15min":3.9020000000000001},"last":0.56399999999999995}]},{"osd":2,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":2.085,"5min":2.613,"15min":2.613},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.194,"5min":0.72499999999999998,"15min":0.65800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":7.9809999999999999,"5min":7.9809999999999999,"15min":7.9809999999999999},"last":0.54000000000000004}]},{"osd":3,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.65800000000000003,"15min":0.63400000000000001},"min":{"1min":0.36199999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.68600000000000005,"15min":0.68899999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.8130000000000002,"5min":2.8130000000000002,"15min":2.8130000000000002},"last":0.65100000000000002}]},{"osd":5,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.42799999999999999,"15min":0.46100000000000002},"min":{"1min":0.25600000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.373},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.438,"15min":0.47299999999999998},"min":{"1min":0.254,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61399999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.41099999999999998}]},{"osd":6,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.442,"15min":0.47999999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.628,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.48599999999999999,"15min":0.53400000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.63600000000000001,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.434}]},{"osd":7,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.48099999999999998,"15min":0.50900000000000001},"min":{"1min":0.308,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.499},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.54400000000000004,"15min":0.60399999999999998},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70499999999999996,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.54700000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574904,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39916,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331924,"statfs":{"total":96636764160,"available":96595890176,"internally_reserved":0,"allocated":626688,"data_stored":448810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40237512},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.57899999999999996,"15min":0.49199999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.8449999999999998,"5min":4.8449999999999998,"15min":4.8449999999999998},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.5529999999999999,"5min":0.77800000000000002,"15min":0.59799999999999998},"min":{"1min":0.41099999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":7.7489999999999997,"5min":7.7489999999999997,"15min":7.7489999999999997},"last":0.42599999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":2.0529999999999999,"5min":0.84299999999999997,"15min":0.60499999999999998},"min":{"1min":0.317,"5min":0.252,"15min":0.252},"max":{"1min":28.908999999999999,"5min":28.908999999999999,"15min":28.908999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":0.73299999999999998,"15min":0.64600000000000002},"min":{"1min":0.40600000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.1520000000000001,"5min":3.1699999999999999,"15min":3.1699999999999999},"last":0.45300000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":0.80300000000000005,"15min":0.63500000000000001},"min":{"1min":0.41799999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":4.3630000000000004,"5min":4.3630000000000004,"15min":4.3630000000000004},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.76900000000000002,"15min":0.69299999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.1859999999999999,"5min":3.2610000000000001,"15min":3.2610000000000001},"last":0.60499999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":0.94299999999999995,"15min":0.82699999999999996},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":4.1029999999999998,"5min":4.1029999999999998,"15min":4.1029999999999998},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.79300000000000004,"15min":0.753},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.903,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":0.754}]},{"osd":5,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":0.80700000000000005,"15min":0.754},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":2.012,"5min":2.5790000000000002,"15min":2.5790000000000002},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.1160000000000001,"5min":0.89100000000000001,"15min":0.84699999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.0739999999999998,"5min":3.1970000000000001,"15min":3.1970000000000001},"last":0.71699999999999997}]},{"osd":6,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":0.96699999999999997,"15min":0.84199999999999997},"min":{"1min":0.61099999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":4.1449999999999996,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.0760000000000001,"5min":0.86499999999999999,"15min":0.82799999999999996},"min":{"1min":0.53000000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.9319999999999999,"5min":2.7360000000000002,"15min":2.7360000000000002},"last":0.79800000000000004}]},{"osd":7,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":0.88500000000000001,"15min":0.82399999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":3.117,"5min":3.2320000000000002,"15min":3.2320000000000002},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.613,"5min":1.006,"15min":0.874},"min":{"1min":0.56599999999999995,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":4.181,"5min":4.181,"15min":4.181},"last":0.86399999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35464,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336376,"statfs":{"total":96636764160,"available":96600449024,"internally_reserved":0,"allocated":262144,"data_stored":84385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.988,"5min":0.996,"15min":0.65800000000000003},"min":{"1min":0.26600000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":31.440999999999999,"5min":31.440999999999999,"15min":31.440999999999999},"last":0.437},{"interface":"front","average":{"1min":1.294,"5min":0.69099999999999995,"15min":0.57699999999999996},"min":{"1min":0.34899999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.7720000000000002,"5min":4.7720000000000002,"15min":4.7720000000000002},"last":0.47199999999999998}]},{"osd":1,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.2949999999999999,"5min":0.89800000000000002,"15min":0.65600000000000003},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.497},{"interface":"front","average":{"1min":1.121,"5min":0.755,"15min":0.61299999999999999},"min":{"1min":0.379,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.0230000000000001,"5min":6.5659999999999998,"15min":6.5659999999999998},"last":0.59099999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":0.71499999999999997,"15min":0.59299999999999997},"min":{"1min":0.371,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":6.5469999999999997,"5min":6.5469999999999997,"15min":6.5469999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.7869999999999999,"5min":0.84499999999999997,"15min":0.68300000000000005},"min":{"1min":0.45500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":8.8659999999999997,"5min":8.8659999999999997,"15min":8.8659999999999997},"last":0.57899999999999996}]},{"osd":4,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":0.81499999999999995,"15min":0.752},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":3.4460000000000002,"5min":3.4460000000000002,"15min":3.4460000000000002},"last":0.63},{"interface":"front","average":{"1min":1.163,"5min":0.80600000000000005,"15min":0.76300000000000001},"min":{"1min":0.46200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":6.4379999999999997,"5min":6.4379999999999997,"15min":6.4379999999999997},"last":0.76800000000000002}]},{"osd":5,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.929,"5min":0.97599999999999998,"15min":0.81699999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":9.1039999999999992,"5min":9.1039999999999992,"15min":9.1039999999999992},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.1899999999999999,"5min":0.84399999999999997,"15min":0.77400000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":3.633,"5min":3.633,"15min":3.633},"last":0.75}]},{"osd":6,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":0.83199999999999996,"15min":0.79400000000000004},"min":{"1min":0.54400000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.95,"5min":0.999,"15min":0.85899999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":9.141,"5min":9.141,"15min":9.141},"last":0.79200000000000004}]},{"osd":7,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":0.88400000000000001,"15min":0.84299999999999997},"min":{"1min":0.50700000000000001,"5min":0.505,"15min":0.505},"max":{"1min":3.5990000000000002,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.2450000000000001,"5min":0.86699999999999999,"15min":0.83499999999999996},"min":{"1min":0.57599999999999996,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":6.4800000000000004,"5min":6.4800000000000004,"15min":6.4800000000000004},"last":0.84199999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35460,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336380,"statfs":{"total":96636764160,"available":96600453120,"internally_reserved":0,"allocated":258048,"data_stored":84564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5630000000000002,"5min":1.0009999999999999,"15min":0.65200000000000002},"min":{"1min":0.29399999999999998,"5min":0.249,"15min":0.249},"max":{"1min":22.721,"5min":22.721,"15min":22.721},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.226,"5min":0.86699999999999999,"15min":0.64100000000000001},"min":{"1min":0.32200000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.2430000000000003,"5min":19.494,"15min":19.494},"last":0.69699999999999995}]},{"osd":2,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3570000000000002,"5min":0.98999999999999999,"15min":0.67200000000000004},"min":{"1min":0.34799999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":15.491,"5min":15.491,"15min":15.491},"last":0.57999999999999996},{"interface":"front","average":{"1min":1.5549999999999999,"5min":0.93300000000000005,"15min":0.70399999999999996},"min":{"1min":0.39600000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.6829999999999998,"5min":17.071000000000002,"15min":17.071000000000002},"last":0.50800000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.631,"5min":0.98599999999999999,"15min":0.73499999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":6.883,"5min":18.838000000000001,"15min":18.838000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.339,"5min":0.92200000000000004,"15min":0.69699999999999995},"min":{"1min":0.40899999999999997,"5min":0.31,"15min":0.31},"max":{"1min":7.0339999999999998,"5min":20.199000000000002,"15min":20.199000000000002},"last":0.63}]},{"osd":4,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.778,"5min":1.2390000000000001,"15min":0.91300000000000003},"min":{"1min":0.39800000000000002,"5min":0.373,"15min":0.373},"max":{"1min":15.654999999999999,"5min":15.654999999999999,"15min":15.654999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.091,"15min":0.89500000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":7.5659999999999998,"5min":18.742000000000001,"15min":18.742000000000001},"last":0.82099999999999995}]},{"osd":5,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":0.97699999999999998,"15min":0.89300000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":7.3230000000000004,"5min":7.3230000000000004,"15min":7.3230000000000004},"last":0.84599999999999997},{"interface":"front","average":{"1min":2.819,"5min":1.296,"15min":0.98299999999999998},"min":{"1min":0.51400000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":15.686,"5min":15.686,"15min":15.686},"last":0.87}]},{"osd":6,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":0.96799999999999997,"15min":0.86599999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":7.3700000000000001,"5min":7.3700000000000001,"15min":7.3700000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":2.851,"5min":1.3180000000000001,"15min":1.002},"min":{"1min":0.55500000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":15.718,"5min":15.718,"15min":15.718},"last":0.83799999999999997}]},{"osd":7,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":1.119,"15min":0.89500000000000002},"min":{"1min":0.495,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":8.1539999999999999,"5min":18.798999999999999,"15min":18.798999999999999},"last":0.88},{"interface":"front","average":{"1min":1.4339999999999999,"5min":0.995,"15min":0.90200000000000002},"min":{"1min":0.53000000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":7.407,"5min":7.4950000000000001,"15min":7.4950000000000001},"last":0.76000000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574904,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39616,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332224,"statfs":{"total":96636764160,"available":96596197376,"internally_reserved":0,"allocated":319488,"data_stored":673303,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.65800000000000003,"15min":0.53000000000000003},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.9239999999999999,"5min":7.2539999999999996,"15min":7.2539999999999996},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.2589999999999999,"5min":0.71999999999999997,"15min":0.55700000000000005},"min":{"1min":0.32400000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":6.0949999999999998,"5min":6.0949999999999998,"15min":6.0949999999999998},"last":0.68899999999999995}]},{"osd":2,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.78300000000000003,"15min":0.627},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.1800000000000002,"5min":7.1920000000000002,"15min":7.1920000000000002},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.151,"5min":0.74299999999999999,"15min":0.59899999999999998},"min":{"1min":0.36399999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":6.5,"5min":6.5,"15min":6.5},"last":0.51200000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":0.66500000000000004,"15min":0.54800000000000004},"min":{"1min":0.38800000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":4.4459999999999997,"5min":4.4459999999999997,"15min":4.4459999999999997},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.85999999999999999,"15min":0.69099999999999995},"min":{"1min":0.38700000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0409999999999999,"5min":10.522,"15min":10.522},"last":0.59799999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":0.89300000000000002,"15min":0.80100000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":6.4560000000000004,"5min":6.4560000000000004,"15min":6.4560000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.82799999999999996,"15min":0.77900000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":3.79,"5min":3.79,"15min":3.79},"last":0.76000000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":0.85399999999999998,"15min":0.78800000000000003},"min":{"1min":0.51300000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.8340000000000001,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":1.002,"5min":0.96599999999999997,"15min":0.85399999999999998},"min":{"1min":0.48599999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":4.5620000000000003,"5min":7.8339999999999996,"15min":7.8339999999999996},"last":0.70799999999999996}]},{"osd":6,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.98199999999999998,"15min":0.86499999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.7270000000000003,"5min":7.5510000000000002,"15min":7.5510000000000002},"last":0.754},{"interface":"front","average":{"1min":1.345,"5min":0.95699999999999996,"15min":0.81499999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":7.444,"5min":7.444,"15min":7.444},"last":0.78000000000000003}]},{"osd":7,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":0.95399999999999996,"15min":0.81599999999999995},"min":{"1min":0.51900000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":7.5350000000000001,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.1639999999999999,"5min":0.874,"15min":0.79700000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.1879999999999997,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":365730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":389120,"data_stored":365450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:48:24.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:48:24.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.213+0000 7f4506c00640 1 Processor -- start 2024-09-13T16:48:24.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.213+0000 7f4506c00640 1 -- start start 2024-09-13T16:48:24.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.214+0000 7f4506c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45000c5bd0 0x7f4500156150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:24.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.214+0000 7f4506c00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f45000c3670 0x7f45000c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:24.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.214+0000 7f4506c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f45000c1ea0 0x7f45000c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:24.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.214+0000 7f4506c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45000c2dc0 con 0x7f45000c1ea0 2024-09-13T16:48:24.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.214+0000 7f4506c00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4500155780 con 0x7f45000c3670 2024-09-13T16:48:24.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.214+0000 7f4506c00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45000c4400 con 0x7f45000c5bd0 2024-09-13T16:48:24.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.214+0000 7f4504c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f45000c1ea0 0x7f45000c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:24.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.214+0000 7f4504c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f45000c1ea0 0x7f45000c6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39166/0 (socket says 172.21.15.28:39166) 2024-09-13T16:48:24.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.214+0000 7f4504c00640 1 -- 172.21.15.28:0/1605421738 learned_addr learned my addr 172.21.15.28:0/1605421738 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:24.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.215+0000 7f44ff400640 1 --2- 172.21.15.28:0/1605421738 >> v2:172.21.15.191:3300/0 conn(0x7f45000c3670 0x7f45000c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:24.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.215+0000 7f44ffe00640 1 --2- 172.21.15.28:0/1605421738 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45000c5bd0 0x7f4500156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:24.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.215+0000 7f4504c00640 1 -- 172.21.15.28:0/1605421738 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45000c5bd0 msgr2=0x7f4500156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:24.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.215+0000 7f4504c00640 1 --2- 172.21.15.28:0/1605421738 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45000c5bd0 0x7f4500156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:24.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.215+0000 7f4504c00640 1 -- 172.21.15.28:0/1605421738 >> v2:172.21.15.191:3300/0 conn(0x7f45000c3670 msgr2=0x7f45000c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:24.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.215+0000 7f4504c00640 1 --2- 172.21.15.28:0/1605421738 >> v2:172.21.15.191:3300/0 conn(0x7f45000c3670 0x7f45000c1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:24.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.215+0000 7f4504c00640 1 -- 172.21.15.28:0/1605421738 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450014ca00 con 0x7f45000c1ea0 2024-09-13T16:48:24.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.216+0000 7f4504c00640 1 --2- 172.21.15.28:0/1605421738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45000c1ea0 0x7f45000c6ef0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f45000c7d70 tx=0x7f44f802a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a784ec8f7186060 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:24.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.216+0000 7f44fea00640 1 -- 172.21.15.28:0/1605421738 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f8031600 con 0x7f45000c1ea0 2024-09-13T16:48:24.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.216+0000 7f44fea00640 1 -- 172.21.15.28:0/1605421738 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f802d050 con 0x7f45000c1ea0 2024-09-13T16:48:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.216+0000 7f44fea00640 1 -- 172.21.15.28:0/1605421738 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f8031980 con 0x7f45000c1ea0 2024-09-13T16:48:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.217+0000 7f4506c00640 1 -- 172.21.15.28:0/1605421738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45000c1ea0 msgr2=0x7f45000c6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.217+0000 7f4506c00640 1 --2- 172.21.15.28:0/1605421738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45000c1ea0 0x7f45000c6ef0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f45000c7d70 tx=0x7f44f802a750 comp rx=0 tx=0).stop 2024-09-13T16:48:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.217+0000 7f4506c00640 1 -- 172.21.15.28:0/1605421738 shutdown_connections 2024-09-13T16:48:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.217+0000 7f4506c00640 1 --2- 172.21.15.28:0/1605421738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45000c1ea0 0x7f45000c6ef0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.217+0000 7f4506c00640 1 --2- 172.21.15.28:0/1605421738 >> v2:172.21.15.191:3300/0 conn(0x7f45000c3670 0x7f45000c1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.217+0000 7f4506c00640 1 --2- 172.21.15.28:0/1605421738 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45000c5bd0 0x7f4500156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.217+0000 7f4506c00640 1 -- 172.21.15.28:0/1605421738 >> 172.21.15.28:0/1605421738 conn(0x7f4500066040 msgr2=0x7f4500069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.217+0000 7f4506c00640 1 -- 172.21.15.28:0/1605421738 shutdown_connections 2024-09-13T16:48:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.217+0000 7f4506c00640 1 -- 172.21.15.28:0/1605421738 wait complete. 2024-09-13T16:48:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.218+0000 7f4506c00640 1 Processor -- start 2024-09-13T16:48:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.219+0000 7f4506c00640 1 -- start start 2024-09-13T16:48:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.219+0000 7f4506c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45000c1ea0 0x7f4500146c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.219+0000 7f4506c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45000c3670 0x7f450014bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.219+0000 7f4506c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45000c5bd0 0x7f4500145940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.219+0000 7f4506c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4500159620 con 0x7f45000c3670 2024-09-13T16:48:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.219+0000 7f4506c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45001ea700 con 0x7f45000c1ea0 2024-09-13T16:48:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.219+0000 7f4506c00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4500147140 con 0x7f45000c5bd0 2024-09-13T16:48:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.220+0000 7f44ffe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45000c1ea0 0x7f4500146c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.220+0000 7f44ffe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45000c1ea0 0x7f4500146c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:38712/0 (socket says 172.21.15.28:38712) 2024-09-13T16:48:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.220+0000 7f44ffe00640 1 -- 172.21.15.28:0/1766033885 learned_addr learned my addr 172.21.15.28:0/1766033885 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.220+0000 7f44ff400640 1 --2- 172.21.15.28:0/1766033885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45000c3670 0x7f450014bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.220+0000 7f4504c00640 1 --2- 172.21.15.28:0/1766033885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45000c5bd0 0x7f4500145940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.220+0000 7f44ffe00640 1 -- 172.21.15.28:0/1766033885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45000c5bd0 msgr2=0x7f4500145940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.220+0000 7f44ffe00640 1 --2- 172.21.15.28:0/1766033885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45000c5bd0 0x7f4500145940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.220+0000 7f44ffe00640 1 -- 172.21.15.28:0/1766033885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45000c3670 msgr2=0x7f450014bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.220+0000 7f44ffe00640 1 --2- 172.21.15.28:0/1766033885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45000c3670 0x7f450014bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.220+0000 7f44ffe00640 1 -- 172.21.15.28:0/1766033885 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f8009b30 con 0x7f45000c1ea0 2024-09-13T16:48:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.220+0000 7f4504c00640 1 --2- 172.21.15.28:0/1766033885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45000c5bd0 0x7f4500145940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.220+0000 7f44ffe00640 1 --2- 172.21.15.28:0/1766033885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45000c1ea0 0x7f4500146c00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f44f000e8a0 tx=0x7f44f000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.221+0000 7f44f7e00640 1 -- 172.21.15.28:0/1766033885 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f00094f0 con 0x7f45000c1ea0 2024-09-13T16:48:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.221+0000 7f44f7e00640 1 -- 172.21.15.28:0/1766033885 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f000c940 con 0x7f45000c1ea0 2024-09-13T16:48:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.221+0000 7f44f7e00640 1 -- 172.21.15.28:0/1766033885 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f000ccf0 con 0x7f45000c1ea0 2024-09-13T16:48:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.221+0000 7f4506c00640 1 -- 172.21.15.28:0/1766033885 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4500144ba0 con 0x7f45000c1ea0 2024-09-13T16:48:24.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.221+0000 7f4506c00640 1 -- 172.21.15.28:0/1766033885 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45001ea8f0 con 0x7f45000c1ea0 2024-09-13T16:48:24.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.222+0000 7f4506c00640 1 -- 172.21.15.28:0/1766033885 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4500070680 con 0x7f45000c1ea0 2024-09-13T16:48:24.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.223+0000 7f44f7e00640 1 -- 172.21.15.28:0/1766033885 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f44f0013470 con 0x7f45000c1ea0 2024-09-13T16:48:24.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.224+0000 7f44f7e00640 1 --2- 172.21.15.28:0/1766033885 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f44e006ac00 0x7f44e006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:24.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.224+0000 7f44f7e00640 1 -- 172.21.15.28:0/1766033885 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f44f0089390 con 0x7f45000c1ea0 2024-09-13T16:48:24.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.224+0000 7f44ff400640 1 --2- 172.21.15.28:0/1766033885 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f44e006ac00 0x7f44e006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:24.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.225+0000 7f44ff400640 1 --2- 172.21.15.28:0/1766033885 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f44e006ac00 0x7f44e006d0c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f450006a190 tx=0x7f44ec009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.228+0000 7f44f7e00640 1 -- 172.21.15.28:0/1766033885 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f44f0054590 con 0x7f45000c1ea0 2024-09-13T16:48:24.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.407+0000 7f4506c00640 1 -- 172.21.15.28:0/1766033885 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45000c45f0 con 0x7f45000c1ea0 2024-09-13T16:48:24.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.409+0000 7f44f7e00640 1 -- 172.21.15.28:0/1766033885 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16673 (secure 0 0 0) 0x7f44f0057bb0 con 0x7f45000c1ea0 2024-09-13T16:48:24.411 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:48:24.411 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":22,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:44:58.950267+0000","last_up_change":"2024-09-13T16:43:49.476313+0000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:43:49.643606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:43:52.828510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:43:53.945874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6801","nonce":2964323435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6803","nonce":2964323435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6807","nonce":2964323435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6805","nonce":2964323435}]},"public_addr":"172.21.15.28:6801/2964323435","cluster_addr":"172.21.15.28:6803/2964323435","heartbeat_back_addr":"172.21.15.28:6807/2964323435","heartbeat_front_addr":"172.21.15.28:6805/2964323435","state":["exists","up"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6809","nonce":23768570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6811","nonce":23768570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6815","nonce":23768570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6813","nonce":23768570}]},"public_addr":"172.21.15.28:6809/23768570","cluster_addr":"172.21.15.28:6811/23768570","heartbeat_back_addr":"172.21.15.28:6815/23768570","heartbeat_front_addr":"172.21.15.28:6813/23768570","state":["exists","up"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6817","nonce":886887303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6819","nonce":886887303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6823","nonce":886887303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6821","nonce":886887303}]},"public_addr":"172.21.15.28:6817/886887303","cluster_addr":"172.21.15.28:6819/886887303","heartbeat_back_addr":"172.21.15.28:6823/886887303","heartbeat_front_addr":"172.21.15.28:6821/886887303","state":["exists","up"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6825","nonce":2616036674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6827","nonce":2616036674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6831","nonce":2616036674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6829","nonce":2616036674}]},"public_addr":"172.21.15.28:6825/2616036674","cluster_addr":"172.21.15.28:6827/2616036674","heartbeat_back_addr":"172.21.15.28:6831/2616036674","heartbeat_front_addr":"172.21.15.28:6829/2616036674","state":["exists","up"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6801","nonce":2817102450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6803","nonce":2817102450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6807","nonce":2817102450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6805","nonce":2817102450}]},"public_addr":"172.21.15.191:6801/2817102450","cluster_addr":"172.21.15.191:6803/2817102450","heartbeat_back_addr":"172.21.15.191:6807/2817102450","heartbeat_front_addr":"172.21.15.191:6805/2817102450","state":["exists","up"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6817","nonce":2503027971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6819","nonce":2503027971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6823","nonce":2503027971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6821","nonce":2503027971}]},"public_addr":"172.21.15.191:6817/2503027971","cluster_addr":"172.21.15.191:6819/2503027971","heartbeat_back_addr":"172.21.15.191:6823/2503027971","heartbeat_front_addr":"172.21.15.191:6821/2503027971","state":["exists","up"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6809","nonce":3006595088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6811","nonce":3006595088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6815","nonce":3006595088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6813","nonce":3006595088}]},"public_addr":"172.21.15.191:6809/3006595088","cluster_addr":"172.21.15.191:6811/3006595088","heartbeat_back_addr":"172.21.15.191:6815/3006595088","heartbeat_front_addr":"172.21.15.191:6813/3006595088","state":["exists","up"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6825","nonce":1477327720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6827","nonce":1477327720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6831","nonce":1477327720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6829","nonce":1477327720}]},"public_addr":"172.21.15.191:6825/1477327720","cluster_addr":"172.21.15.191:6827/1477327720","heartbeat_back_addr":"172.21.15.191:6831/1477327720","heartbeat_front_addr":"172.21.15.191:6829/1477327720","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.917250+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.954351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.017329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.046221+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.707467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.726947+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.782321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.764850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:48:24.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.411+0000 7f4506c00640 1 -- 172.21.15.28:0/1766033885 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f44e006ac00 msgr2=0x7f44e006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:24.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.411+0000 7f4506c00640 1 --2- 172.21.15.28:0/1766033885 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f44e006ac00 0x7f44e006d0c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f450006a190 tx=0x7f44ec009190 comp rx=0 tx=0).stop 2024-09-13T16:48:24.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.411+0000 7f4506c00640 1 -- 172.21.15.28:0/1766033885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45000c1ea0 msgr2=0x7f4500146c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:24.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.411+0000 7f4506c00640 1 --2- 172.21.15.28:0/1766033885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45000c1ea0 0x7f4500146c00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f44f000e8a0 tx=0x7f44f000ed60 comp rx=0 tx=0).stop 2024-09-13T16:48:24.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.411+0000 7f4506c00640 1 -- 172.21.15.28:0/1766033885 shutdown_connections 2024-09-13T16:48:24.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.411+0000 7f4506c00640 1 --2- 172.21.15.28:0/1766033885 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f44e006ac00 0x7f44e006d0c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:24.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.411+0000 7f4506c00640 1 --2- 172.21.15.28:0/1766033885 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f45000c5bd0 0x7f4500145940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:24.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.411+0000 7f4506c00640 1 --2- 172.21.15.28:0/1766033885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45000c3670 0x7f450014bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:24.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.411+0000 7f4506c00640 1 --2- 172.21.15.28:0/1766033885 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45000c1ea0 0x7f4500146c00 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:24.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.411+0000 7f4506c00640 1 -- 172.21.15.28:0/1766033885 >> 172.21.15.28:0/1766033885 conn(0x7f4500066040 msgr2=0x7f45000c6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:24.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.411+0000 7f4506c00640 1 -- 172.21.15.28:0/1766033885 shutdown_connections 2024-09-13T16:48:24.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:24.411+0000 7f4506c00640 1 -- 172.21.15.28:0/1766033885 wait complete. 2024-09-13T16:48:25.430 INFO:tasks.ceph:Scrubbing osd.0 2024-09-13T16:48:25.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-13T16:48:25.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.599+0000 7fa774200640 1 Processor -- start 2024-09-13T16:48:25.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.599+0000 7fa774200640 1 -- start start 2024-09-13T16:48:25.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.600+0000 7fa774200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa76c0c5bd0 0x7fa76c156150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.600+0000 7fa774200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa76c0c3670 0x7fa76c0c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.600+0000 7fa774200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa76c0c1ea0 0x7fa76c0c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.600+0000 7fa774200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa76c0c2dc0 con 0x7fa76c0c5bd0 2024-09-13T16:48:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.600+0000 7fa774200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa76c155780 con 0x7fa76c0c3670 2024-09-13T16:48:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.601+0000 7fa774200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c0c4400 con 0x7fa76c0c1ea0 2024-09-13T16:48:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.601+0000 7fa771800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa76c0c5bd0 0x7fa76c156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.601+0000 7fa771800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa76c0c5bd0 0x7fa76c156150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39186/0 (socket says 172.21.15.28:39186) 2024-09-13T16:48:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.601+0000 7fa771800640 1 -- 172.21.15.28:0/904007664 learned_addr learned my addr 172.21.15.28:0/904007664 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.601+0000 7fa770e00640 1 --2- 172.21.15.28:0/904007664 >> v2:172.21.15.191:3300/0 conn(0x7fa76c0c3670 0x7fa76c0c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.601+0000 7fa772200640 1 --2- 172.21.15.28:0/904007664 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa76c0c1ea0 0x7fa76c0c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.601+0000 7fa771800640 1 -- 172.21.15.28:0/904007664 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa76c0c1ea0 msgr2=0x7fa76c0c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.601+0000 7fa771800640 1 --2- 172.21.15.28:0/904007664 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa76c0c1ea0 0x7fa76c0c6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.601+0000 7fa771800640 1 -- 172.21.15.28:0/904007664 >> v2:172.21.15.191:3300/0 conn(0x7fa76c0c3670 msgr2=0x7fa76c0c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:25.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.601+0000 7fa771800640 1 --2- 172.21.15.28:0/904007664 >> v2:172.21.15.191:3300/0 conn(0x7fa76c0c3670 0x7fa76c0c1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.601+0000 7fa771800640 1 -- 172.21.15.28:0/904007664 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c14ca00 con 0x7fa76c0c5bd0 2024-09-13T16:48:25.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.602+0000 7fa771800640 1 --2- 172.21.15.28:0/904007664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c0c5bd0 0x7fa76c156150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa76c0c1830 tx=0x7fa76002a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c7237f9abaa81b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:25.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.603+0000 7fa76be00640 1 -- 172.21.15.28:0/904007664 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760032440 con 0x7fa76c0c5bd0 2024-09-13T16:48:25.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.603+0000 7fa76be00640 1 -- 172.21.15.28:0/904007664 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7600325a0 con 0x7fa76c0c5bd0 2024-09-13T16:48:25.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.603+0000 7fa76be00640 1 -- 172.21.15.28:0/904007664 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa760032930 con 0x7fa76c0c5bd0 2024-09-13T16:48:25.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.603+0000 7fa774200640 1 -- 172.21.15.28:0/904007664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c0c5bd0 msgr2=0x7fa76c156150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:25.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.603+0000 7fa774200640 1 --2- 172.21.15.28:0/904007664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c0c5bd0 0x7fa76c156150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa76c0c1830 tx=0x7fa76002a2a0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.603+0000 7fa774200640 1 -- 172.21.15.28:0/904007664 shutdown_connections 2024-09-13T16:48:25.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.603+0000 7fa774200640 1 --2- 172.21.15.28:0/904007664 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa76c0c1ea0 0x7fa76c0c6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.603+0000 7fa774200640 1 --2- 172.21.15.28:0/904007664 >> v2:172.21.15.191:3300/0 conn(0x7fa76c0c3670 0x7fa76c0c1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.603+0000 7fa774200640 1 --2- 172.21.15.28:0/904007664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c0c5bd0 0x7fa76c156150 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.603+0000 7fa774200640 1 -- 172.21.15.28:0/904007664 >> 172.21.15.28:0/904007664 conn(0x7fa76c066040 msgr2=0x7fa76c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.603+0000 7fa774200640 1 -- 172.21.15.28:0/904007664 shutdown_connections 2024-09-13T16:48:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.603+0000 7fa774200640 1 -- 172.21.15.28:0/904007664 wait complete. 2024-09-13T16:48:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.605+0000 7fa774200640 1 Processor -- start 2024-09-13T16:48:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.605+0000 7fa774200640 1 -- start start 2024-09-13T16:48:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa774200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c0c1ea0 0x7fa76c151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa774200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa76c0c3670 0x7fa76c1494a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa774200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa76c0c5bd0 0x7fa76c0c71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa774200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c1eac40 con 0x7fa76c0c1ea0 2024-09-13T16:48:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa774200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c1ea370 con 0x7fa76c0c3670 2024-09-13T16:48:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa774200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c152530 con 0x7fa76c0c5bd0 2024-09-13T16:48:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa771800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c0c1ea0 0x7fa76c151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa771800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c0c1ea0 0x7fa76c151e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39192/0 (socket says 172.21.15.28:39192) 2024-09-13T16:48:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa771800640 1 -- 172.21.15.28:0/1481478978 learned_addr learned my addr 172.21.15.28:0/1481478978 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa770e00640 1 --2- 172.21.15.28:0/1481478978 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa76c0c3670 0x7fa76c1494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa772200640 1 --2- 172.21.15.28:0/1481478978 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa76c0c5bd0 0x7fa76c0c71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa771800640 1 -- 172.21.15.28:0/1481478978 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa76c0c5bd0 msgr2=0x7fa76c0c71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa771800640 1 --2- 172.21.15.28:0/1481478978 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa76c0c5bd0 0x7fa76c0c71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa771800640 1 -- 172.21.15.28:0/1481478978 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa76c0c3670 msgr2=0x7fa76c1494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa771800640 1 --2- 172.21.15.28:0/1481478978 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa76c0c3670 0x7fa76c1494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa771800640 1 -- 172.21.15.28:0/1481478978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7600094d0 con 0x7fa76c0c1ea0 2024-09-13T16:48:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa770e00640 1 --2- 172.21.15.28:0/1481478978 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa76c0c3670 0x7fa76c1494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:48:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.606+0000 7fa772200640 1 --2- 172.21.15.28:0/1481478978 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa76c0c5bd0 0x7fa76c0c71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:48:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.607+0000 7fa771800640 1 --2- 172.21.15.28:0/1481478978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c0c1ea0 0x7fa76c151e80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa7600097f0 tx=0x7fa7600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.607+0000 7fa769600640 1 -- 172.21.15.28:0/1481478978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76003d830 con 0x7fa76c0c1ea0 2024-09-13T16:48:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.607+0000 7fa769600640 1 -- 172.21.15.28:0/1481478978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76003d990 con 0x7fa76c0c1ea0 2024-09-13T16:48:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.607+0000 7fa769600640 1 -- 172.21.15.28:0/1481478978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76003dd40 con 0x7fa76c0c1ea0 2024-09-13T16:48:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.607+0000 7fa774200640 1 -- 172.21.15.28:0/1481478978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76c152810 con 0x7fa76c0c1ea0 2024-09-13T16:48:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.607+0000 7fa774200640 1 -- 172.21.15.28:0/1481478978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76c1ea860 con 0x7fa76c0c1ea0 2024-09-13T16:48:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.608+0000 7fa774200640 1 -- 172.21.15.28:0/1481478978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa76c0c5070 con 0x7fa76c0c1ea0 2024-09-13T16:48:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.609+0000 7fa769600640 1 -- 172.21.15.28:0/1481478978 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa760039040 con 0x7fa76c0c1ea0 2024-09-13T16:48:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.610+0000 7fa769600640 1 --2- 172.21.15.28:0/1481478978 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa74c06ac30 0x7fa74c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.610+0000 7fa769600640 1 -- 172.21.15.28:0/1481478978 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa7600a9ee0 con 0x7fa76c0c1ea0 2024-09-13T16:48:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.610+0000 7fa769600640 1 --2- 172.21.15.28:0/1481478978 >> [v2:172.21.15.28:6800/2964323435,v1:172.21.15.28:6801/2964323435] conn(0x7fa74c072f50 0x7fa74c0753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.610+0000 7fa769600640 1 -- 172.21.15.28:0/1481478978 --> [v2:172.21.15.28:6800/2964323435,v1:172.21.15.28:6801/2964323435] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa74c075aa0 con 0x7fa74c072f50 2024-09-13T16:48:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.610+0000 7fa769600640 1 -- 172.21.15.28:0/1481478978 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7600ac4c0 con 0x7fa76c0c1ea0 2024-09-13T16:48:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.610+0000 7fa770e00640 1 --2- 172.21.15.28:0/1481478978 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa74c06ac30 0x7fa74c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.610+0000 7fa772200640 1 --2- 172.21.15.28:0/1481478978 >> [v2:172.21.15.28:6800/2964323435,v1:172.21.15.28:6801/2964323435] conn(0x7fa74c072f50 0x7fa74c0753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.611+0000 7fa770e00640 1 --2- 172.21.15.28:0/1481478978 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa74c06ac30 0x7fa74c06d0f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa76c1e1df0 tx=0x7fa758008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.611+0000 7fa772200640 1 --2- 172.21.15.28:0/1481478978 >> [v2:172.21.15.28:6800/2964323435,v1:172.21.15.28:6801/2964323435] conn(0x7fa74c072f50 0x7fa74c0753d0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.612+0000 7fa769600640 1 -- 172.21.15.28:0/1481478978 <== osd.0 v2:172.21.15.28:6800/2964323435 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fa74c075aa0 con 0x7fa74c072f50 2024-09-13T16:48:25.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.624+0000 7fa774200640 1 -- 172.21.15.28:0/1481478978 --> [v2:172.21.15.28:6800/2964323435,v1:172.21.15.28:6801/2964323435] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa76c0c3e40 con 0x7fa74c072f50 2024-09-13T16:48:25.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.625+0000 7fa769600640 1 -- 172.21.15.28:0/1481478978 <== osd.0 v2:172.21.15.28:6800/2964323435 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fa76400f040 con 0x7fa74c072f50 2024-09-13T16:48:25.627 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:48:25.627 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T16:48:25.627 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:48:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.625+0000 7fa774200640 1 -- 172.21.15.28:0/1481478978 >> [v2:172.21.15.28:6800/2964323435,v1:172.21.15.28:6801/2964323435] conn(0x7fa74c072f50 msgr2=0x7fa74c0753d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.625+0000 7fa774200640 1 --2- 172.21.15.28:0/1481478978 >> [v2:172.21.15.28:6800/2964323435,v1:172.21.15.28:6801/2964323435] conn(0x7fa74c072f50 0x7fa74c0753d0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.625+0000 7fa774200640 1 -- 172.21.15.28:0/1481478978 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa74c06ac30 msgr2=0x7fa74c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.625+0000 7fa774200640 1 --2- 172.21.15.28:0/1481478978 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa74c06ac30 0x7fa74c06d0f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa76c1e1df0 tx=0x7fa758008040 comp rx=0 tx=0).stop 2024-09-13T16:48:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.625+0000 7fa774200640 1 -- 172.21.15.28:0/1481478978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c0c1ea0 msgr2=0x7fa76c151e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:25.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.625+0000 7fa774200640 1 --2- 172.21.15.28:0/1481478978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa76c0c1ea0 0x7fa76c151e80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa7600097f0 tx=0x7fa7600046c0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.625+0000 7fa772200640 1 -- 172.21.15.28:0/1481478978 reap_dead start 2024-09-13T16:48:25.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.625+0000 7fa774200640 1 -- 172.21.15.28:0/1481478978 shutdown_connections 2024-09-13T16:48:25.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.625+0000 7fa774200640 1 -- 172.21.15.28:0/1481478978 >> 172.21.15.28:0/1481478978 conn(0x7fa76c066040 msgr2=0x7fa76c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:25.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.625+0000 7fa774200640 1 -- 172.21.15.28:0/1481478978 shutdown_connections 2024-09-13T16:48:25.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.625+0000 7fa774200640 1 -- 172.21.15.28:0/1481478978 wait complete. 2024-09-13T16:48:25.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-13T16:48:25.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.797+0000 7ffbca400640 1 Processor -- start 2024-09-13T16:48:25.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.797+0000 7ffbca400640 1 -- start start 2024-09-13T16:48:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.798+0000 7ffbca400640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ffbc40c5140 0x7ffbc4159800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.798+0000 7ffbca400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffbc40c73e0 0x7ffbc40c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.798+0000 7ffbca400640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc40c6f50 con 0x7ffbc40c5a60 2024-09-13T16:48:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.798+0000 7ffbca400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc40c6940 con 0x7ffbc40c73e0 2024-09-13T16:48:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.798+0000 7ffbca400640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc4159d40 con 0x7ffbc40c5140 2024-09-13T16:48:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.798+0000 7ffbc2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffbc40c73e0 0x7ffbc40c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.798+0000 7ffbc2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffbc40c73e0 0x7ffbc40c5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39204/0 (socket says 172.21.15.28:39204) 2024-09-13T16:48:25.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.798+0000 7ffbc2a00640 1 -- 172.21.15.28:0/3757791091 learned_addr learned my addr 172.21.15.28:0/3757791091 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:25.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.798+0000 7ffbc3400640 1 --2- 172.21.15.28:0/3757791091 >> v2:172.21.15.191:3300/0 conn(0x7ffbc40c5140 0x7ffbc4159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:25.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.799+0000 7ffbc2a00640 1 -- 172.21.15.28:0/3757791091 >> v2:172.21.15.191:3300/0 conn(0x7ffbc40c5140 msgr2=0x7ffbc4159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:25.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.799+0000 7ffbc2a00640 1 --2- 172.21.15.28:0/3757791091 >> v2:172.21.15.191:3300/0 conn(0x7ffbc40c5140 0x7ffbc4159800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.799+0000 7ffbc2a00640 1 -- 172.21.15.28:0/3757791091 >> v1:172.21.15.191:6789/0 conn(0x7ffbc40c5a60 legacy=0x7ffbc414f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:25.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.799+0000 7ffbc2a00640 1 -- 172.21.15.28:0/3757791091 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc40c1b50 con 0x7ffbc40c73e0 2024-09-13T16:48:25.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.799+0000 7ffbc3400640 1 --2- 172.21.15.28:0/3757791091 >> v2:172.21.15.191:3300/0 conn(0x7ffbc40c5140 0x7ffbc4159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:25.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.800+0000 7ffbc2a00640 1 --2- 172.21.15.28:0/3757791091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc40c73e0 0x7ffbc40c5520 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ffbc406ea30 tx=0x7ffbb002a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=302d20c804d28687 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:25.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.800+0000 7ffbc2000640 1 -- 172.21.15.28:0/3757791091 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0032440 con 0x7ffbc40c73e0 2024-09-13T16:48:25.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.800+0000 7ffbc2000640 1 -- 172.21.15.28:0/3757791091 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb00325a0 con 0x7ffbc40c73e0 2024-09-13T16:48:25.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.800+0000 7ffbc2000640 1 -- 172.21.15.28:0/3757791091 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0032930 con 0x7ffbc40c73e0 2024-09-13T16:48:25.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.801+0000 7ffbca400640 1 -- 172.21.15.28:0/3757791091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc40c73e0 msgr2=0x7ffbc40c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:25.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.801+0000 7ffbca400640 1 --2- 172.21.15.28:0/3757791091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc40c73e0 0x7ffbc40c5520 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ffbc406ea30 tx=0x7ffbb002a200 comp rx=0 tx=0).stop 2024-09-13T16:48:25.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.801+0000 7ffbca400640 1 -- 172.21.15.28:0/3757791091 shutdown_connections 2024-09-13T16:48:25.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.801+0000 7ffbca400640 1 --2- 172.21.15.28:0/3757791091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc40c73e0 0x7ffbc40c5520 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.801+0000 7ffbca400640 1 --2- 172.21.15.28:0/3757791091 >> v2:172.21.15.191:3300/0 conn(0x7ffbc40c5140 0x7ffbc4159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.801+0000 7ffbca400640 1 -- 172.21.15.28:0/3757791091 >> 172.21.15.28:0/3757791091 conn(0x7ffbc40ca6b0 msgr2=0x7ffbc40caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:25.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.801+0000 7ffbca400640 1 -- 172.21.15.28:0/3757791091 shutdown_connections 2024-09-13T16:48:25.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.801+0000 7ffbca400640 1 -- 172.21.15.28:0/3757791091 wait complete. 2024-09-13T16:48:25.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.802+0000 7ffbca400640 1 Processor -- start 2024-09-13T16:48:25.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.803+0000 7ffbca400640 1 -- start start 2024-09-13T16:48:25.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.803+0000 7ffbca400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc40c5140 0x7ffbc41f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:25.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.803+0000 7ffbca400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbc40c5a60 0x7ffbc41450e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:25.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.803+0000 7ffbca400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbc40c73e0 0x7ffbc414ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:25.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.803+0000 7ffbca400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc41f29c0 con 0x7ffbc40c5140 2024-09-13T16:48:25.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.803+0000 7ffbca400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc4159390 con 0x7ffbc40c5a60 2024-09-13T16:48:25.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.803+0000 7ffbca400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc4159670 con 0x7ffbc40c73e0 2024-09-13T16:48:25.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.804+0000 7ffbc3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc40c5140 0x7ffbc41f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:25.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.804+0000 7ffbc3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc40c5140 0x7ffbc41f2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39212/0 (socket says 172.21.15.28:39212) 2024-09-13T16:48:25.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.804+0000 7ffbc3400640 1 -- 172.21.15.28:0/2221054597 learned_addr learned my addr 172.21.15.28:0/2221054597 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:25.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.804+0000 7ffbc2a00640 1 --2- 172.21.15.28:0/2221054597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbc40c5a60 0x7ffbc41450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:25.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.804+0000 7ffbc3e00640 1 --2- 172.21.15.28:0/2221054597 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbc40c73e0 0x7ffbc414ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.804+0000 7ffbc3400640 1 -- 172.21.15.28:0/2221054597 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbc40c73e0 msgr2=0x7ffbc414ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.804+0000 7ffbc3400640 1 --2- 172.21.15.28:0/2221054597 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbc40c73e0 0x7ffbc414ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.804+0000 7ffbc3400640 1 -- 172.21.15.28:0/2221054597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbc40c5a60 msgr2=0x7ffbc41450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.804+0000 7ffbc3400640 1 --2- 172.21.15.28:0/2221054597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbc40c5a60 0x7ffbc41450e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.804+0000 7ffbc3400640 1 -- 172.21.15.28:0/2221054597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb400b6a0 con 0x7ffbc40c5140 2024-09-13T16:48:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.804+0000 7ffbc2a00640 1 --2- 172.21.15.28:0/2221054597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbc40c5a60 0x7ffbc41450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.804+0000 7ffbc3400640 1 --2- 172.21.15.28:0/2221054597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc40c5140 0x7ffbc41f2480 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffbb400ed40 tx=0x7ffbb400d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.805+0000 7ffbbb400640 1 -- 172.21.15.28:0/2221054597 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4010640 con 0x7ffbc40c5140 2024-09-13T16:48:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.805+0000 7ffbbb400640 1 -- 172.21.15.28:0/2221054597 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb40107a0 con 0x7ffbc40c5140 2024-09-13T16:48:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.805+0000 7ffbbb400640 1 -- 172.21.15.28:0/2221054597 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4010b10 con 0x7ffbc40c5140 2024-09-13T16:48:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.805+0000 7ffbca400640 1 -- 172.21.15.28:0/2221054597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb00094d0 con 0x7ffbc40c5140 2024-09-13T16:48:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.805+0000 7ffbca400640 1 -- 172.21.15.28:0/2221054597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc41f1c80 con 0x7ffbc40c5140 2024-09-13T16:48:25.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.806+0000 7ffbca400640 1 -- 172.21.15.28:0/2221054597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc4159b90 con 0x7ffbc40c5140 2024-09-13T16:48:25.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.807+0000 7ffbbb400640 1 -- 172.21.15.28:0/2221054597 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ffbb4010c70 con 0x7ffbc40c5140 2024-09-13T16:48:25.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.807+0000 7ffbbb400640 1 --2- 172.21.15.28:0/2221054597 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ffba406ac30 0x7ffba406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:25.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.808+0000 7ffbbb400640 1 -- 172.21.15.28:0/2221054597 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ffbb4089ab0 con 0x7ffbc40c5140 2024-09-13T16:48:25.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.808+0000 7ffbc2a00640 1 --2- 172.21.15.28:0/2221054597 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ffba406ac30 0x7ffba406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:25.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.809+0000 7ffbc2a00640 1 --2- 172.21.15.28:0/2221054597 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ffba406ac30 0x7ffba406d0f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0034000 tx=0x7ffbb002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:25.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.812+0000 7ffbbb400640 1 -- 172.21.15.28:0/2221054597 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ffbb4054b90 con 0x7ffbc40c5140 2024-09-13T16:48:25.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.989+0000 7ffbca400640 1 -- 172.21.15.28:0/2221054597 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7ffbc41f1b00 con 0x7ffba406ac30 2024-09-13T16:48:25.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.990+0000 7ffbbb400640 1 -- 172.21.15.28:0/2221054597 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ffbb0032dd0 con 0x7ffba406ac30 2024-09-13T16:48:25.991 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 0 to deep-scrub 2024-09-13T16:48:25.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.992+0000 7ffbca400640 1 -- 172.21.15.28:0/2221054597 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ffba406ac30 msgr2=0x7ffba406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:25.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.992+0000 7ffbca400640 1 --2- 172.21.15.28:0/2221054597 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ffba406ac30 0x7ffba406d0f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0034000 tx=0x7ffbb002e000 comp rx=0 tx=0).stop 2024-09-13T16:48:25.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.992+0000 7ffbca400640 1 -- 172.21.15.28:0/2221054597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc40c5140 msgr2=0x7ffbc41f2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:25.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.992+0000 7ffbca400640 1 --2- 172.21.15.28:0/2221054597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc40c5140 0x7ffbc41f2480 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffbb400ed40 tx=0x7ffbb400d720 comp rx=0 tx=0).stop 2024-09-13T16:48:25.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.992+0000 7ffbca400640 1 -- 172.21.15.28:0/2221054597 shutdown_connections 2024-09-13T16:48:25.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.992+0000 7ffbca400640 1 --2- 172.21.15.28:0/2221054597 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ffba406ac30 0x7ffba406d0f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.992+0000 7ffbca400640 1 --2- 172.21.15.28:0/2221054597 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffbc40c73e0 0x7ffbc414ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.992+0000 7ffbca400640 1 --2- 172.21.15.28:0/2221054597 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffbc40c5a60 0x7ffbc41450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.992+0000 7ffbca400640 1 --2- 172.21.15.28:0/2221054597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc40c5140 0x7ffbc41f2480 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:25.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.992+0000 7ffbca400640 1 -- 172.21.15.28:0/2221054597 >> 172.21.15.28:0/2221054597 conn(0x7ffbc40ca6b0 msgr2=0x7ffbc40c4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:25.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.992+0000 7ffbca400640 1 -- 172.21.15.28:0/2221054597 shutdown_connections 2024-09-13T16:48:25.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:25.992+0000 7ffbca400640 1 -- 172.21.15.28:0/2221054597 wait complete. 2024-09-13T16:48:26.010 INFO:tasks.ceph:Scrubbing osd.1 2024-09-13T16:48:26.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-13T16:48:26.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.168+0000 7ff993800640 1 Processor -- start 2024-09-13T16:48:26.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.168+0000 7ff993800640 1 -- start start 2024-09-13T16:48:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.170+0000 7ff993800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff98c150ca0 0x7ff98c1568e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.170+0000 7ff993800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff98c0c2250 0x7ff98c15a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.170+0000 7ff993800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff98c151080 con 0x7ff98c0c4d60 2024-09-13T16:48:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.170+0000 7ff993800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff98c0c2670 con 0x7ff98c0c2250 2024-09-13T16:48:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.170+0000 7ff993800640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff98c06a990 con 0x7ff98c150ca0 2024-09-13T16:48:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.170+0000 7ff98be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff98c0c2250 0x7ff98c15a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.170+0000 7ff991800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff98c0c4d60 0x7ff98c06d8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52738/0 (socket says 172.21.15.28:52738) 2024-09-13T16:48:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.170+0000 7ff991800640 1 -- 172.21.15.28:0/1002706725 learned_addr learned my addr 172.21.15.28:0/1002706725 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:26.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.170+0000 7ff990e00640 1 --2- 172.21.15.28:0/1002706725 >> v2:172.21.15.191:3300/0 conn(0x7ff98c150ca0 0x7ff98c1568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.170+0000 7ff990e00640 1 -- 172.21.15.28:0/1002706725 >> v2:172.21.15.28:3300/0 conn(0x7ff98c0c2250 msgr2=0x7ff98c15a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:26.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.170+0000 7ff990e00640 1 --2- 172.21.15.28:0/1002706725 >> v2:172.21.15.28:3300/0 conn(0x7ff98c0c2250 0x7ff98c15a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.170+0000 7ff990e00640 1 -- 172.21.15.28:0/1002706725 >> v1:172.21.15.28:6789/0 conn(0x7ff98c0c4d60 legacy=0x7ff98c06d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.170+0000 7ff990e00640 1 -- 172.21.15.28:0/1002706725 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98c0c2870 con 0x7ff98c150ca0 2024-09-13T16:48:26.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.172+0000 7ff990e00640 1 --2- 172.21.15.28:0/1002706725 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff98c150ca0 0x7ff98c1568e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff98c072100 tx=0x7ff98002a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=755ceb19f92c0425 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:26.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.172+0000 7ff98b400640 1 -- 172.21.15.28:0/1002706725 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98002ae60 con 0x7ff98c150ca0 2024-09-13T16:48:26.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.172+0000 7ff98b400640 1 -- 172.21.15.28:0/1002706725 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff980032440 con 0x7ff98c150ca0 2024-09-13T16:48:26.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.172+0000 7ff98b400640 1 -- 172.21.15.28:0/1002706725 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9800327d0 con 0x7ff98c150ca0 2024-09-13T16:48:26.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.172+0000 7ff993800640 1 -- 172.21.15.28:0/1002706725 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff98c150ca0 msgr2=0x7ff98c1568e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.172+0000 7ff993800640 1 --2- 172.21.15.28:0/1002706725 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff98c150ca0 0x7ff98c1568e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff98c072100 tx=0x7ff98002a110 comp rx=0 tx=0).stop 2024-09-13T16:48:26.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.173+0000 7ff993800640 1 -- 172.21.15.28:0/1002706725 shutdown_connections 2024-09-13T16:48:26.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.173+0000 7ff993800640 1 --2- 172.21.15.28:0/1002706725 >> v2:172.21.15.28:3300/0 conn(0x7ff98c0c2250 0x7ff98c15a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.173+0000 7ff993800640 1 --2- 172.21.15.28:0/1002706725 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff98c150ca0 0x7ff98c1568e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.173+0000 7ff993800640 1 -- 172.21.15.28:0/1002706725 >> 172.21.15.28:0/1002706725 conn(0x7ff98c066040 msgr2=0x7ff98c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:26.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.173+0000 7ff993800640 1 -- 172.21.15.28:0/1002706725 shutdown_connections 2024-09-13T16:48:26.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.173+0000 7ff993800640 1 -- 172.21.15.28:0/1002706725 wait complete. 2024-09-13T16:48:26.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.174+0000 7ff993800640 1 Processor -- start 2024-09-13T16:48:26.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.174+0000 7ff993800640 1 -- start start 2024-09-13T16:48:26.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.175+0000 7ff993800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98c0c2250 0x7ff98c0704d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.175+0000 7ff993800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff98c0c4d60 0x7ff98c070a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.175+0000 7ff993800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff98c150ca0 0x7ff98c149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.175+0000 7ff993800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98c157040 con 0x7ff98c0c2250 2024-09-13T16:48:26.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.175+0000 7ff993800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98c14b4d0 con 0x7ff98c150ca0 2024-09-13T16:48:26.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.175+0000 7ff993800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98c14bce0 con 0x7ff98c0c4d60 2024-09-13T16:48:26.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.175+0000 7ff990e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98c0c2250 0x7ff98c0704d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.176+0000 7ff990e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98c0c2250 0x7ff98c0704d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39224/0 (socket says 172.21.15.28:39224) 2024-09-13T16:48:26.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.176+0000 7ff990e00640 1 -- 172.21.15.28:0/3648774509 learned_addr learned my addr 172.21.15.28:0/3648774509 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:26.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.176+0000 7ff991800640 1 --2- 172.21.15.28:0/3648774509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff98c150ca0 0x7ff98c149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.176+0000 7ff98be00640 1 --2- 172.21.15.28:0/3648774509 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff98c0c4d60 0x7ff98c070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.176+0000 7ff990e00640 1 -- 172.21.15.28:0/3648774509 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff98c0c4d60 msgr2=0x7ff98c070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.176+0000 7ff990e00640 1 --2- 172.21.15.28:0/3648774509 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff98c0c4d60 0x7ff98c070a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.176+0000 7ff990e00640 1 -- 172.21.15.28:0/3648774509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff98c150ca0 msgr2=0x7ff98c149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.176+0000 7ff990e00640 1 --2- 172.21.15.28:0/3648774509 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff98c150ca0 0x7ff98c149f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.176+0000 7ff990e00640 1 -- 172.21.15.28:0/3648774509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff980009400 con 0x7ff98c0c2250 2024-09-13T16:48:26.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.176+0000 7ff990e00640 1 --2- 172.21.15.28:0/3648774509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98c0c2250 0x7ff98c0704d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff980000c00 tx=0x7ff9800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:26.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.177+0000 7ff988c00640 1 -- 172.21.15.28:0/3648774509 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98003d8f0 con 0x7ff98c0c2250 2024-09-13T16:48:26.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.177+0000 7ff988c00640 1 -- 172.21.15.28:0/3648774509 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98003da70 con 0x7ff98c0c2250 2024-09-13T16:48:26.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.177+0000 7ff988c00640 1 -- 172.21.15.28:0/3648774509 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98003de20 con 0x7ff98c0c2250 2024-09-13T16:48:26.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.177+0000 7ff993800640 1 -- 172.21.15.28:0/3648774509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98c1eee20 con 0x7ff98c0c2250 2024-09-13T16:48:26.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.177+0000 7ff993800640 1 -- 172.21.15.28:0/3648774509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98c14b720 con 0x7ff98c0c2250 2024-09-13T16:48:26.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.178+0000 7ff993800640 1 -- 172.21.15.28:0/3648774509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff98c159f30 con 0x7ff98c0c2250 2024-09-13T16:48:26.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.179+0000 7ff988c00640 1 -- 172.21.15.28:0/3648774509 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff980039040 con 0x7ff98c0c2250 2024-09-13T16:48:26.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.179+0000 7ff988c00640 1 --2- 172.21.15.28:0/3648774509 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff96c06ac80 0x7ff96c06d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.180+0000 7ff988c00640 1 -- 172.21.15.28:0/3648774509 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff9800a9c60 con 0x7ff98c0c2250 2024-09-13T16:48:26.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.180+0000 7ff988c00640 1 --2- 172.21.15.28:0/3648774509 >> [v2:172.21.15.28:6808/23768570,v1:172.21.15.28:6809/23768570] conn(0x7ff96c073020 0x7ff96c0754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.180+0000 7ff988c00640 1 -- 172.21.15.28:0/3648774509 --> [v2:172.21.15.28:6808/23768570,v1:172.21.15.28:6809/23768570] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff96c075b70 con 0x7ff96c073020 2024-09-13T16:48:26.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.180+0000 7ff988c00640 1 -- 172.21.15.28:0/3648774509 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9800ac370 con 0x7ff98c0c2250 2024-09-13T16:48:26.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.180+0000 7ff98be00640 1 --2- 172.21.15.28:0/3648774509 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff96c06ac80 0x7ff96c06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.180+0000 7ff991800640 1 --2- 172.21.15.28:0/3648774509 >> [v2:172.21.15.28:6808/23768570,v1:172.21.15.28:6809/23768570] conn(0x7ff96c073020 0x7ff96c0754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.181+0000 7ff98be00640 1 --2- 172.21.15.28:0/3648774509 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff96c06ac80 0x7ff96c06d140 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff98c14e030 tx=0x7ff978009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:26.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.181+0000 7ff991800640 1 --2- 172.21.15.28:0/3648774509 >> [v2:172.21.15.28:6808/23768570,v1:172.21.15.28:6809/23768570] conn(0x7ff96c073020 0x7ff96c0754a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:26.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.181+0000 7ff988c00640 1 -- 172.21.15.28:0/3648774509 <== osd.1 v2:172.21.15.28:6808/23768570 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7ff96c075b70 con 0x7ff96c073020 2024-09-13T16:48:26.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.194+0000 7ff993800640 1 -- 172.21.15.28:0/3648774509 --> [v2:172.21.15.28:6808/23768570,v1:172.21.15.28:6809/23768570] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff98c0c2e30 con 0x7ff96c073020 2024-09-13T16:48:26.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.194+0000 7ff988c00640 1 -- 172.21.15.28:0/3648774509 <== osd.1 v2:172.21.15.28:6808/23768570 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff984001f20 con 0x7ff96c073020 2024-09-13T16:48:26.196 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:48:26.196 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T16:48:26.196 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:48:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.195+0000 7ff993800640 1 -- 172.21.15.28:0/3648774509 >> [v2:172.21.15.28:6808/23768570,v1:172.21.15.28:6809/23768570] conn(0x7ff96c073020 msgr2=0x7ff96c0754a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.195+0000 7ff993800640 1 --2- 172.21.15.28:0/3648774509 >> [v2:172.21.15.28:6808/23768570,v1:172.21.15.28:6809/23768570] conn(0x7ff96c073020 0x7ff96c0754a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.195+0000 7ff993800640 1 -- 172.21.15.28:0/3648774509 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff96c06ac80 msgr2=0x7ff96c06d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.195+0000 7ff993800640 1 --2- 172.21.15.28:0/3648774509 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff96c06ac80 0x7ff96c06d140 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff98c14e030 tx=0x7ff978009190 comp rx=0 tx=0).stop 2024-09-13T16:48:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.195+0000 7ff993800640 1 -- 172.21.15.28:0/3648774509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98c0c2250 msgr2=0x7ff98c0704d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.195+0000 7ff993800640 1 --2- 172.21.15.28:0/3648774509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff98c0c2250 0x7ff98c0704d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff980000c00 tx=0x7ff9800046c0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.195+0000 7ff991800640 1 -- 172.21.15.28:0/3648774509 reap_dead start 2024-09-13T16:48:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.195+0000 7ff993800640 1 -- 172.21.15.28:0/3648774509 shutdown_connections 2024-09-13T16:48:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.195+0000 7ff993800640 1 -- 172.21.15.28:0/3648774509 >> 172.21.15.28:0/3648774509 conn(0x7ff98c066040 msgr2=0x7ff98c14f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.195+0000 7ff993800640 1 -- 172.21.15.28:0/3648774509 shutdown_connections 2024-09-13T16:48:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.195+0000 7ff993800640 1 -- 172.21.15.28:0/3648774509 wait complete. 2024-09-13T16:48:26.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-13T16:48:26.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.367+0000 7ff1f0200640 1 Processor -- start 2024-09-13T16:48:26.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.367+0000 7ff1f0200640 1 -- start start 2024-09-13T16:48:26.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.368+0000 7ff1f0200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff1e80c67b0 0x7ff1e806d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.368+0000 7ff1f0200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1e806d770 0x7ff1e806fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.368+0000 7ff1f0200640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e815cbc0 con 0x7ff1e8152d50 2024-09-13T16:48:26.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.368+0000 7ff1f0200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1e815c080 con 0x7ff1e80c67b0 2024-09-13T16:48:26.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.368+0000 7ff1f0200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e80c5a50 con 0x7ff1e806d770 2024-09-13T16:48:26.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.368+0000 7ff1ece00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff1e80c67b0 0x7ff1e806d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.368+0000 7ff1ee200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1e806d770 0x7ff1e806fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.368+0000 7ff1ece00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff1e80c67b0 0x7ff1e806d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:38748/0 (socket says 172.21.15.28:38748) 2024-09-13T16:48:26.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.368+0000 7ff1ece00640 1 -- 172.21.15.28:0/1415841875 learned_addr learned my addr 172.21.15.28:0/1415841875 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:26.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.369+0000 7ff1ece00640 1 -- 172.21.15.28:0/1415841875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1e806d770 msgr2=0x7ff1e806fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:26.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.369+0000 7ff1ece00640 1 --2- 172.21.15.28:0/1415841875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1e806d770 0x7ff1e806fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.369+0000 7ff1ece00640 1 -- 172.21.15.28:0/1415841875 >> v1:172.21.15.191:6789/0 conn(0x7ff1e8152d50 legacy=0x7ff1e80c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.369+0000 7ff1ece00640 1 -- 172.21.15.28:0/1415841875 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e80c5d10 con 0x7ff1e80c67b0 2024-09-13T16:48:26.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.369+0000 7ff1e7e00640 1 -- 172.21.15.28:0/1415841875 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038905832 0 0) 0x7ff1e815cbc0 con 0x7ff1e8152d50 2024-09-13T16:48:26.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.370+0000 7ff1ece00640 1 --2- 172.21.15.28:0/1415841875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1e80c67b0 0x7ff1e806d230 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4009870 tx=0x7ff1d402a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eabdd2347d736399 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:26.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.370+0000 7ff1e7e00640 1 -- 172.21.15.28:0/1415841875 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4032640 con 0x7ff1e80c67b0 2024-09-13T16:48:26.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.370+0000 7ff1e7e00640 1 -- 172.21.15.28:0/1415841875 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d40327a0 con 0x7ff1e80c67b0 2024-09-13T16:48:26.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.370+0000 7ff1e7e00640 1 -- 172.21.15.28:0/1415841875 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4032b30 con 0x7ff1e80c67b0 2024-09-13T16:48:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.371+0000 7ff1f0200640 1 -- 172.21.15.28:0/1415841875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1e80c67b0 msgr2=0x7ff1e806d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.371+0000 7ff1f0200640 1 --2- 172.21.15.28:0/1415841875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1e80c67b0 0x7ff1e806d230 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4009870 tx=0x7ff1d402a300 comp rx=0 tx=0).stop 2024-09-13T16:48:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.371+0000 7ff1f0200640 1 -- 172.21.15.28:0/1415841875 shutdown_connections 2024-09-13T16:48:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.371+0000 7ff1f0200640 1 --2- 172.21.15.28:0/1415841875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1e806d770 0x7ff1e806fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.371+0000 7ff1f0200640 1 --2- 172.21.15.28:0/1415841875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1e80c67b0 0x7ff1e806d230 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.371+0000 7ff1f0200640 1 -- 172.21.15.28:0/1415841875 >> 172.21.15.28:0/1415841875 conn(0x7ff1e80ca6b0 msgr2=0x7ff1e80caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.371+0000 7ff1f0200640 1 -- 172.21.15.28:0/1415841875 shutdown_connections 2024-09-13T16:48:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.371+0000 7ff1f0200640 1 -- 172.21.15.28:0/1415841875 wait complete. 2024-09-13T16:48:26.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.372+0000 7ff1f0200640 1 Processor -- start 2024-09-13T16:48:26.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.372+0000 7ff1f0200640 1 -- start start 2024-09-13T16:48:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.373+0000 7ff1f0200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1e806d770 0x7ff1e814dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.373+0000 7ff1f0200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1e80c67b0 0x7ff1e814c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.373+0000 7ff1f0200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1e8152d50 0x7ff1e81ee3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.373+0000 7ff1f0200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e8149e50 con 0x7ff1e806d770 2024-09-13T16:48:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.373+0000 7ff1f0200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e814a0f0 con 0x7ff1e8152d50 2024-09-13T16:48:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.373+0000 7ff1f0200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e814d090 con 0x7ff1e80c67b0 2024-09-13T16:48:26.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.374+0000 7ff1ee200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1e8152d50 0x7ff1e81ee3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.374+0000 7ff1ece00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1e80c67b0 0x7ff1e814c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.374+0000 7ff1ed800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1e806d770 0x7ff1e814dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.374+0000 7ff1ee200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1e8152d50 0x7ff1e81ee3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:38760/0 (socket says 172.21.15.28:38760) 2024-09-13T16:48:26.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.374+0000 7ff1ee200640 1 -- 172.21.15.28:0/1819686817 learned_addr learned my addr 172.21.15.28:0/1819686817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:26.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.374+0000 7ff1ee200640 1 -- 172.21.15.28:0/1819686817 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1e80c67b0 msgr2=0x7ff1e814c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.374+0000 7ff1ee200640 1 --2- 172.21.15.28:0/1819686817 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1e80c67b0 0x7ff1e814c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.374+0000 7ff1ee200640 1 -- 172.21.15.28:0/1819686817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1e806d770 msgr2=0x7ff1e814dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.374+0000 7ff1ee200640 1 --2- 172.21.15.28:0/1819686817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1e806d770 0x7ff1e814dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.374+0000 7ff1ee200640 1 -- 172.21.15.28:0/1819686817 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d40094d0 con 0x7ff1e8152d50 2024-09-13T16:48:26.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.374+0000 7ff1ece00640 1 --2- 172.21.15.28:0/1819686817 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1e80c67b0 0x7ff1e814c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:26.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.374+0000 7ff1ed800640 1 --2- 172.21.15.28:0/1819686817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1e806d770 0x7ff1e814dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:26.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.374+0000 7ff1ee200640 1 --2- 172.21.15.28:0/1819686817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1e8152d50 0x7ff1e81ee3a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff1e000efc0 tx=0x7ff1e000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:26.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.375+0000 7ff1e5600640 1 -- 172.21.15.28:0/1819686817 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0014480 con 0x7ff1e8152d50 2024-09-13T16:48:26.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.375+0000 7ff1e5600640 1 -- 172.21.15.28:0/1819686817 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e000f040 con 0x7ff1e8152d50 2024-09-13T16:48:26.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.375+0000 7ff1e5600640 1 -- 172.21.15.28:0/1819686817 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0014820 con 0x7ff1e8152d50 2024-09-13T16:48:26.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.375+0000 7ff1f0200640 1 -- 172.21.15.28:0/1819686817 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e814d300 con 0x7ff1e8152d50 2024-09-13T16:48:26.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.375+0000 7ff1f0200640 1 -- 172.21.15.28:0/1819686817 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e814b530 con 0x7ff1e8152d50 2024-09-13T16:48:26.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.376+0000 7ff1f0200640 1 -- 172.21.15.28:0/1819686817 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e8153130 con 0x7ff1e8152d50 2024-09-13T16:48:26.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.377+0000 7ff1e5600640 1 -- 172.21.15.28:0/1819686817 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff1e0014980 con 0x7ff1e8152d50 2024-09-13T16:48:26.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.378+0000 7ff1e5600640 1 --2- 172.21.15.28:0/1819686817 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff1c806ac50 0x7ff1c806d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.378+0000 7ff1e5600640 1 -- 172.21.15.28:0/1819686817 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff1e0089dd0 con 0x7ff1e8152d50 2024-09-13T16:48:26.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.378+0000 7ff1ed800640 1 --2- 172.21.15.28:0/1819686817 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff1c806ac50 0x7ff1c806d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.378+0000 7ff1ed800640 1 --2- 172.21.15.28:0/1819686817 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff1c806ac50 0x7ff1c806d110 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc000f60 tx=0x7ff1dc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:26.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.382+0000 7ff1e5600640 1 -- 172.21.15.28:0/1819686817 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff1e0055050 con 0x7ff1e8152d50 2024-09-13T16:48:26.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.560+0000 7ff1f0200640 1 -- 172.21.15.28:0/1819686817 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1e80c4c20 con 0x7ff1c806ac50 2024-09-13T16:48:26.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.561+0000 7ff1e5600640 1 -- 172.21.15.28:0/1819686817 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff1e80c4c20 con 0x7ff1c806ac50 2024-09-13T16:48:26.562 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 1 to deep-scrub 2024-09-13T16:48:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.563+0000 7ff1f0200640 1 -- 172.21.15.28:0/1819686817 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff1c806ac50 msgr2=0x7ff1c806d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.563+0000 7ff1f0200640 1 --2- 172.21.15.28:0/1819686817 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff1c806ac50 0x7ff1c806d110 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc000f60 tx=0x7ff1dc009040 comp rx=0 tx=0).stop 2024-09-13T16:48:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.563+0000 7ff1f0200640 1 -- 172.21.15.28:0/1819686817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1e8152d50 msgr2=0x7ff1e81ee3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.563+0000 7ff1f0200640 1 --2- 172.21.15.28:0/1819686817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1e8152d50 0x7ff1e81ee3a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff1e000efc0 tx=0x7ff1e000c490 comp rx=0 tx=0).stop 2024-09-13T16:48:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.563+0000 7ff1f0200640 1 -- 172.21.15.28:0/1819686817 shutdown_connections 2024-09-13T16:48:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.563+0000 7ff1f0200640 1 --2- 172.21.15.28:0/1819686817 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff1c806ac50 0x7ff1c806d110 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.563+0000 7ff1f0200640 1 --2- 172.21.15.28:0/1819686817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1e8152d50 0x7ff1e81ee3a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.563+0000 7ff1f0200640 1 --2- 172.21.15.28:0/1819686817 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1e80c67b0 0x7ff1e814c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.563+0000 7ff1f0200640 1 --2- 172.21.15.28:0/1819686817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1e806d770 0x7ff1e814dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.563+0000 7ff1f0200640 1 -- 172.21.15.28:0/1819686817 >> 172.21.15.28:0/1819686817 conn(0x7ff1e80ca6b0 msgr2=0x7ff1e80c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.563+0000 7ff1f0200640 1 -- 172.21.15.28:0/1819686817 shutdown_connections 2024-09-13T16:48:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.563+0000 7ff1f0200640 1 -- 172.21.15.28:0/1819686817 wait complete. 2024-09-13T16:48:26.580 INFO:tasks.ceph:Scrubbing osd.2 2024-09-13T16:48:26.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-13T16:48:26.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.739+0000 7f97da000640 1 Processor -- start 2024-09-13T16:48:26.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.739+0000 7f97da000640 1 -- start start 2024-09-13T16:48:26.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.740+0000 7f97da000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f97d4150ca0 0x7f97d41568e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.740+0000 7f97da000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97d40c4d60 0x7f97d40c5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.741+0000 7f97da000640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d4151080 con 0x7f97d40c2250 2024-09-13T16:48:26.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.741+0000 7f97da000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d40c2670 con 0x7f97d40c4d60 2024-09-13T16:48:26.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.741+0000 7f97da000640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d415ae40 con 0x7f97d4150ca0 2024-09-13T16:48:26.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.741+0000 7f97d3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97d40c4d60 0x7f97d40c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.741+0000 7f97d3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97d40c4d60 0x7f97d40c5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39240/0 (socket says 172.21.15.28:39240) 2024-09-13T16:48:26.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.741+0000 7f97d3e00640 1 -- 172.21.15.28:0/3718157333 learned_addr learned my addr 172.21.15.28:0/3718157333 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:26.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.741+0000 7f97d3400640 1 --2- 172.21.15.28:0/3718157333 >> v2:172.21.15.191:3300/0 conn(0x7f97d4150ca0 0x7f97d41568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.741+0000 7f97d3e00640 1 -- 172.21.15.28:0/3718157333 >> v2:172.21.15.191:3300/0 conn(0x7f97d4150ca0 msgr2=0x7f97d41568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:26.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.741+0000 7f97d3e00640 1 --2- 172.21.15.28:0/3718157333 >> v2:172.21.15.191:3300/0 conn(0x7f97d4150ca0 0x7f97d41568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.741+0000 7f97d3e00640 1 -- 172.21.15.28:0/3718157333 >> v1:172.21.15.191:6789/0 conn(0x7f97d40c2250 legacy=0x7f97d415a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.741+0000 7f97d3e00640 1 -- 172.21.15.28:0/3718157333 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d406a990 con 0x7f97d40c4d60 2024-09-13T16:48:26.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.742+0000 7f97d3e00640 1 --2- 172.21.15.28:0/3718157333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d40c4d60 0x7f97d40c5190 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f97d4072100 tx=0x7f97cc02a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604803eeaed5d972 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:26.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.742+0000 7f97d2a00640 1 -- 172.21.15.28:0/3718157333 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97cc031600 con 0x7f97d40c4d60 2024-09-13T16:48:26.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.743+0000 7f97d2a00640 1 -- 172.21.15.28:0/3718157333 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97cc02d050 con 0x7f97d40c4d60 2024-09-13T16:48:26.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.743+0000 7f97d2a00640 1 -- 172.21.15.28:0/3718157333 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97cc031980 con 0x7f97d40c4d60 2024-09-13T16:48:26.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.743+0000 7f97da000640 1 -- 172.21.15.28:0/3718157333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d40c4d60 msgr2=0x7f97d40c5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.743+0000 7f97da000640 1 --2- 172.21.15.28:0/3718157333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d40c4d60 0x7f97d40c5190 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f97d4072100 tx=0x7f97cc02a6d0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.743+0000 7f97da000640 1 -- 172.21.15.28:0/3718157333 shutdown_connections 2024-09-13T16:48:26.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.743+0000 7f97da000640 1 --2- 172.21.15.28:0/3718157333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d40c4d60 0x7f97d40c5190 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.743+0000 7f97da000640 1 --2- 172.21.15.28:0/3718157333 >> v2:172.21.15.191:3300/0 conn(0x7f97d4150ca0 0x7f97d41568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.743+0000 7f97da000640 1 -- 172.21.15.28:0/3718157333 >> 172.21.15.28:0/3718157333 conn(0x7f97d4066040 msgr2=0x7f97d4069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:26.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.743+0000 7f97da000640 1 -- 172.21.15.28:0/3718157333 shutdown_connections 2024-09-13T16:48:26.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.743+0000 7f97da000640 1 -- 172.21.15.28:0/3718157333 wait complete. 2024-09-13T16:48:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.745+0000 7f97da000640 1 Processor -- start 2024-09-13T16:48:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.745+0000 7f97da000640 1 -- start start 2024-09-13T16:48:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.746+0000 7f97da000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97d40c2250 0x7f97d4158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.746+0000 7f97da000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d40c4d60 0x7f97d414be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.746+0000 7f97da000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97d4150ca0 0x7f97d414a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.746+0000 7f97da000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d4158ff0 con 0x7f97d40c4d60 2024-09-13T16:48:26.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.746+0000 7f97da000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d4158d10 con 0x7f97d4150ca0 2024-09-13T16:48:26.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.746+0000 7f97da000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d414b0c0 con 0x7f97d40c2250 2024-09-13T16:48:26.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.746+0000 7f97d3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97d4150ca0 0x7f97d414a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.746+0000 7f97d3400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97d40c2250 0x7f97d4158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.746+0000 7f97d3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97d4150ca0 0x7f97d414a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:38776/0 (socket says 172.21.15.28:38776) 2024-09-13T16:48:26.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.746+0000 7f97d3e00640 1 -- 172.21.15.28:0/2661184541 learned_addr learned my addr 172.21.15.28:0/2661184541 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:26.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.746+0000 7f97cbe00640 1 --2- 172.21.15.28:0/2661184541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d40c4d60 0x7f97d414be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.747+0000 7f97d3400640 1 -- 172.21.15.28:0/2661184541 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97d4150ca0 msgr2=0x7f97d414a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.747+0000 7f97d3400640 1 --2- 172.21.15.28:0/2661184541 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97d4150ca0 0x7f97d414a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.747+0000 7f97d3400640 1 -- 172.21.15.28:0/2661184541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d40c4d60 msgr2=0x7f97d414be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.747+0000 7f97d3400640 1 --2- 172.21.15.28:0/2661184541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d40c4d60 0x7f97d414be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.747+0000 7f97d3400640 1 -- 172.21.15.28:0/2661184541 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97cc009b30 con 0x7f97d40c2250 2024-09-13T16:48:26.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.747+0000 7f97d3e00640 1 --2- 172.21.15.28:0/2661184541 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97d4150ca0 0x7f97d414a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:26.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.747+0000 7f97cbe00640 1 --2- 172.21.15.28:0/2661184541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d40c4d60 0x7f97d414be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:26.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.747+0000 7f97d3400640 1 --2- 172.21.15.28:0/2661184541 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97d40c2250 0x7f97d4158640 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f97c400d520 tx=0x7f97c400d9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:26.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.747+0000 7f97cb400640 1 -- 172.21.15.28:0/2661184541 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c40125f0 con 0x7f97d40c2250 2024-09-13T16:48:26.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.747+0000 7f97cb400640 1 -- 172.21.15.28:0/2661184541 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c4012750 con 0x7f97d40c2250 2024-09-13T16:48:26.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.747+0000 7f97da000640 1 -- 172.21.15.28:0/2661184541 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d414b3a0 con 0x7f97d40c2250 2024-09-13T16:48:26.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.747+0000 7f97cb400640 1 -- 172.21.15.28:0/2661184541 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4012ae0 con 0x7f97d40c2250 2024-09-13T16:48:26.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.747+0000 7f97da000640 1 -- 172.21.15.28:0/2661184541 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d414b960 con 0x7f97d40c2250 2024-09-13T16:48:26.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.748+0000 7f97da000640 1 -- 172.21.15.28:0/2661184541 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f97d40c2e30 con 0x7f97d40c2250 2024-09-13T16:48:26.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.749+0000 7f97cb400640 1 -- 172.21.15.28:0/2661184541 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f97c4012c40 con 0x7f97d40c2250 2024-09-13T16:48:26.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.750+0000 7f97cb400640 1 --2- 172.21.15.28:0/2661184541 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f97b006ac30 0x7f97b006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.750+0000 7f97cb400640 1 -- 172.21.15.28:0/2661184541 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f97c40896f0 con 0x7f97d40c2250 2024-09-13T16:48:26.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.750+0000 7f97cbe00640 1 --2- 172.21.15.28:0/2661184541 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f97b006ac30 0x7f97b006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.750+0000 7f97cb400640 1 --2- 172.21.15.28:0/2661184541 >> [v2:172.21.15.28:6816/886887303,v1:172.21.15.28:6817/886887303] conn(0x7f97b0072f50 0x7f97b00753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.751+0000 7f97cb400640 1 -- 172.21.15.28:0/2661184541 --> [v2:172.21.15.28:6816/886887303,v1:172.21.15.28:6817/886887303] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97b0075aa0 con 0x7f97b0072f50 2024-09-13T16:48:26.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.751+0000 7f97cb400640 1 -- 172.21.15.28:0/2661184541 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97c408bcd0 con 0x7f97d40c2250 2024-09-13T16:48:26.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.751+0000 7f97d3e00640 1 --2- 172.21.15.28:0/2661184541 >> [v2:172.21.15.28:6816/886887303,v1:172.21.15.28:6817/886887303] conn(0x7f97b0072f50 0x7f97b00753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.751+0000 7f97cbe00640 1 --2- 172.21.15.28:0/2661184541 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f97b006ac30 0x7f97b006d0f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f97bc001eb0 tx=0x7f97bc00d040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:26.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.751+0000 7f97d3e00640 1 --2- 172.21.15.28:0/2661184541 >> [v2:172.21.15.28:6816/886887303,v1:172.21.15.28:6817/886887303] conn(0x7f97b0072f50 0x7f97b00753d0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:26.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.752+0000 7f97cb400640 1 -- 172.21.15.28:0/2661184541 <== osd.2 v2:172.21.15.28:6816/886887303 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f97b0075aa0 con 0x7f97b0072f50 2024-09-13T16:48:26.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.765+0000 7f97da000640 1 -- 172.21.15.28:0/2661184541 --> [v2:172.21.15.28:6816/886887303,v1:172.21.15.28:6817/886887303] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f97d40c2b50 con 0x7f97b0072f50 2024-09-13T16:48:26.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.765+0000 7f97cb400640 1 -- 172.21.15.28:0/2661184541 <== osd.2 v2:172.21.15.28:6816/886887303 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f97cc0319b0 con 0x7f97b0072f50 2024-09-13T16:48:26.767 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:48:26.767 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T16:48:26.767 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:48:26.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.765+0000 7f97da000640 1 -- 172.21.15.28:0/2661184541 >> [v2:172.21.15.28:6816/886887303,v1:172.21.15.28:6817/886887303] conn(0x7f97b0072f50 msgr2=0x7f97b00753d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.765+0000 7f97da000640 1 --2- 172.21.15.28:0/2661184541 >> [v2:172.21.15.28:6816/886887303,v1:172.21.15.28:6817/886887303] conn(0x7f97b0072f50 0x7f97b00753d0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.766+0000 7f97da000640 1 -- 172.21.15.28:0/2661184541 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f97b006ac30 msgr2=0x7f97b006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.766+0000 7f97da000640 1 --2- 172.21.15.28:0/2661184541 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f97b006ac30 0x7f97b006d0f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f97bc001eb0 tx=0x7f97bc00d040 comp rx=0 tx=0).stop 2024-09-13T16:48:26.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.766+0000 7f97da000640 1 -- 172.21.15.28:0/2661184541 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97d40c2250 msgr2=0x7f97d4158640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.766+0000 7f97da000640 1 --2- 172.21.15.28:0/2661184541 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97d40c2250 0x7f97d4158640 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f97c400d520 tx=0x7f97c400d9e0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.766+0000 7f97d3e00640 1 -- 172.21.15.28:0/2661184541 reap_dead start 2024-09-13T16:48:26.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.766+0000 7f97da000640 1 -- 172.21.15.28:0/2661184541 shutdown_connections 2024-09-13T16:48:26.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.766+0000 7f97da000640 1 -- 172.21.15.28:0/2661184541 >> 172.21.15.28:0/2661184541 conn(0x7f97d4066040 msgr2=0x7f97d40c6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:26.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.766+0000 7f97da000640 1 -- 172.21.15.28:0/2661184541 shutdown_connections 2024-09-13T16:48:26.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.766+0000 7f97da000640 1 -- 172.21.15.28:0/2661184541 wait complete. 2024-09-13T16:48:26.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-13T16:48:26.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.936+0000 7f4ee7000640 1 Processor -- start 2024-09-13T16:48:26.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.936+0000 7f4ee7000640 1 -- start start 2024-09-13T16:48:26.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.937+0000 7f4ee7000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ee006dcb0 0x7f4ee00c5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.937+0000 7f4ee7000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee00c1d80 con 0x7f4ee015a060 2024-09-13T16:48:26.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.937+0000 7f4ee7000640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee015a920 con 0x7f4ee015a440 2024-09-13T16:48:26.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.937+0000 7f4ee7000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee015ac00 con 0x7f4ee006dcb0 2024-09-13T16:48:26.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.938+0000 7f4edfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4ee015a060 0x7f4ee006d5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52748/0 (socket says 172.21.15.28:52748) 2024-09-13T16:48:26.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.938+0000 7f4edfe00640 1 -- 172.21.15.28:0/997407658 learned_addr learned my addr 172.21.15.28:0/997407658 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:26.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.938+0000 7f4ee5000640 1 --2- 172.21.15.28:0/997407658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ee006dcb0 0x7f4ee00c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.938+0000 7f4edf400640 1 -- 172.21.15.28:0/997407658 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913631547 0 0) 0x7f4ee015a920 con 0x7f4ee015a440 2024-09-13T16:48:26.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.938+0000 7f4edf400640 1 -- 172.21.15.28:0/997407658 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ec4003520 con 0x7f4ee015a440 2024-09-13T16:48:26.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.938+0000 7f4edf400640 1 -- 172.21.15.28:0/997407658 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469880584 0 0) 0x7f4ee00c1d80 con 0x7f4ee015a060 2024-09-13T16:48:26.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.938+0000 7f4edf400640 1 -- 172.21.15.28:0/997407658 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ee015a920 con 0x7f4ee015a060 2024-09-13T16:48:26.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.938+0000 7f4ee5000640 1 -- 172.21.15.28:0/997407658 >> v1:172.21.15.191:6789/0 conn(0x7f4ee015a440 legacy=0x7f4ee01504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.938+0000 7f4ee5000640 1 -- 172.21.15.28:0/997407658 >> v1:172.21.15.28:6789/0 conn(0x7f4ee015a060 legacy=0x7f4ee006d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.938+0000 7f4ee5000640 1 -- 172.21.15.28:0/997407658 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee00c2d50 con 0x7f4ee006dcb0 2024-09-13T16:48:26.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.939+0000 7f4ee5000640 1 --2- 172.21.15.28:0/997407658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ee006dcb0 0x7f4ee00c5500 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009df0 tx=0x7f4ed802a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc6d63777188bc1b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.940+0000 7f4edf400640 1 -- 172.21.15.28:0/997407658 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8007740 con 0x7f4ee006dcb0 2024-09-13T16:48:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.940+0000 7f4edf400640 1 -- 172.21.15.28:0/997407658 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed802d050 con 0x7f4ee006dcb0 2024-09-13T16:48:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.940+0000 7f4edf400640 1 -- 172.21.15.28:0/997407658 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8007ac0 con 0x7f4ee006dcb0 2024-09-13T16:48:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.940+0000 7f4ee7000640 1 -- 172.21.15.28:0/997407658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ee006dcb0 msgr2=0x7f4ee00c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.940+0000 7f4ee7000640 1 --2- 172.21.15.28:0/997407658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ee006dcb0 0x7f4ee00c5500 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009df0 tx=0x7f4ed802a710 comp rx=0 tx=0).stop 2024-09-13T16:48:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.940+0000 7f4ee7000640 1 -- 172.21.15.28:0/997407658 shutdown_connections 2024-09-13T16:48:26.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.940+0000 7f4ee7000640 1 --2- 172.21.15.28:0/997407658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ee006dcb0 0x7f4ee00c5500 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.940+0000 7f4ee7000640 1 -- 172.21.15.28:0/997407658 >> 172.21.15.28:0/997407658 conn(0x7f4ee00ca6b0 msgr2=0x7f4ee00caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:26.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.940+0000 7f4ee7000640 1 -- 172.21.15.28:0/997407658 shutdown_connections 2024-09-13T16:48:26.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.940+0000 7f4ee7000640 1 -- 172.21.15.28:0/997407658 wait complete. 2024-09-13T16:48:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.942+0000 7f4ee7000640 1 Processor -- start 2024-09-13T16:48:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.942+0000 7f4ee7000640 1 -- start start 2024-09-13T16:48:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.943+0000 7f4ee7000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ee006dcb0 0x7f4ee0148b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.943+0000 7f4ee7000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ee015a060 0x7f4ee014c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.943+0000 7f4ee7000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ee015a440 0x7f4ee014c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.943+0000 7f4ee7000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee0159c50 con 0x7f4ee015a440 2024-09-13T16:48:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.943+0000 7f4ee7000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee014af50 con 0x7f4ee006dcb0 2024-09-13T16:48:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.943+0000 7f4ee7000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee014b1c0 con 0x7f4ee015a060 2024-09-13T16:48:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.943+0000 7f4ee5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ee015a440 0x7f4ee014c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.943+0000 7f4ee5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ee015a440 0x7f4ee014c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39254/0 (socket says 172.21.15.28:39254) 2024-09-13T16:48:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.943+0000 7f4ee5000640 1 -- 172.21.15.28:0/438964998 learned_addr learned my addr 172.21.15.28:0/438964998 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.943+0000 7f4edfe00640 1 --2- 172.21.15.28:0/438964998 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ee006dcb0 0x7f4ee0148b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.943+0000 7f4ed7400640 1 --2- 172.21.15.28:0/438964998 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ee015a060 0x7f4ee014c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.944+0000 7f4ee5000640 1 -- 172.21.15.28:0/438964998 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ee015a060 msgr2=0x7f4ee014c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.944+0000 7f4ee5000640 1 --2- 172.21.15.28:0/438964998 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ee015a060 0x7f4ee014c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.944+0000 7f4ee5000640 1 -- 172.21.15.28:0/438964998 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ee006dcb0 msgr2=0x7f4ee0148b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.944+0000 7f4ee5000640 1 --2- 172.21.15.28:0/438964998 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ee006dcb0 0x7f4ee0148b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.944+0000 7f4ee5000640 1 -- 172.21.15.28:0/438964998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed8009a70 con 0x7f4ee015a440 2024-09-13T16:48:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.944+0000 7f4ee5000640 1 --2- 172.21.15.28:0/438964998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ee015a440 0x7f4ee014c5e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009f10 tx=0x7f4ed8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.944+0000 7f4edcc00640 1 -- 172.21.15.28:0/438964998 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed803b820 con 0x7f4ee015a440 2024-09-13T16:48:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.944+0000 7f4edcc00640 1 -- 172.21.15.28:0/438964998 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed803b980 con 0x7f4ee015a440 2024-09-13T16:48:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.944+0000 7f4edcc00640 1 -- 172.21.15.28:0/438964998 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed803bd10 con 0x7f4ee015a440 2024-09-13T16:48:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.944+0000 7f4ee7000640 1 -- 172.21.15.28:0/438964998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee014b4a0 con 0x7f4ee015a440 2024-09-13T16:48:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.944+0000 7f4ee7000640 1 -- 172.21.15.28:0/438964998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee014ba60 con 0x7f4ee015a440 2024-09-13T16:48:26.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.945+0000 7f4ee7000640 1 -- 172.21.15.28:0/438964998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee0067a30 con 0x7f4ee015a440 2024-09-13T16:48:26.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.946+0000 7f4edcc00640 1 -- 172.21.15.28:0/438964998 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4ed8031400 con 0x7f4ee015a440 2024-09-13T16:48:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.947+0000 7f4edcc00640 1 --2- 172.21.15.28:0/438964998 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4ebc06ac00 0x7f4ebc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.947+0000 7f4edcc00640 1 -- 172.21.15.28:0/438964998 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4ed80aa430 con 0x7f4ee015a440 2024-09-13T16:48:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.947+0000 7f4edfe00640 1 --2- 172.21.15.28:0/438964998 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4ebc06ac00 0x7f4ebc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.948+0000 7f4edfe00640 1 --2- 172.21.15.28:0/438964998 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4ebc06ac00 0x7f4ebc06d0c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00011e0 tx=0x7f4ed0007d30 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:26.951+0000 7f4edcc00640 1 -- 172.21.15.28:0/438964998 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4ed8075510 con 0x7f4ee015a440 2024-09-13T16:48:27.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.133+0000 7f4ee7000640 1 -- 172.21.15.28:0/438964998 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ee00c3310 con 0x7f4ebc06ac00 2024-09-13T16:48:27.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.134+0000 7f4edcc00640 1 -- 172.21.15.28:0/438964998 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4ed0005850 con 0x7f4ebc06ac00 2024-09-13T16:48:27.136 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 2 to deep-scrub 2024-09-13T16:48:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.136+0000 7f4ee7000640 1 -- 172.21.15.28:0/438964998 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4ebc06ac00 msgr2=0x7f4ebc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.136+0000 7f4ee7000640 1 --2- 172.21.15.28:0/438964998 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4ebc06ac00 0x7f4ebc06d0c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00011e0 tx=0x7f4ed0007d30 comp rx=0 tx=0).stop 2024-09-13T16:48:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.137+0000 7f4ee7000640 1 -- 172.21.15.28:0/438964998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ee015a440 msgr2=0x7f4ee014c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.137+0000 7f4ee7000640 1 --2- 172.21.15.28:0/438964998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ee015a440 0x7f4ee014c5e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009f10 tx=0x7f4ed8004a40 comp rx=0 tx=0).stop 2024-09-13T16:48:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.137+0000 7f4ee7000640 1 -- 172.21.15.28:0/438964998 shutdown_connections 2024-09-13T16:48:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.137+0000 7f4ee7000640 1 --2- 172.21.15.28:0/438964998 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f4ebc06ac00 0x7f4ebc06d0c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.137+0000 7f4ee7000640 1 --2- 172.21.15.28:0/438964998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ee015a440 0x7f4ee014c5e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.137+0000 7f4ee7000640 1 --2- 172.21.15.28:0/438964998 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ee015a060 0x7f4ee014c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.137+0000 7f4ee7000640 1 --2- 172.21.15.28:0/438964998 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ee006dcb0 0x7f4ee0148b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.137+0000 7f4ee7000640 1 -- 172.21.15.28:0/438964998 >> 172.21.15.28:0/438964998 conn(0x7f4ee00ca6b0 msgr2=0x7f4ee00c64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.137+0000 7f4ee7000640 1 -- 172.21.15.28:0/438964998 shutdown_connections 2024-09-13T16:48:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.137+0000 7f4ee7000640 1 -- 172.21.15.28:0/438964998 wait complete. 2024-09-13T16:48:27.154 INFO:tasks.ceph:Scrubbing osd.3 2024-09-13T16:48:27.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-13T16:48:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.312+0000 7fde17200640 1 Processor -- start 2024-09-13T16:48:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.313+0000 7fde17200640 1 -- start start 2024-09-13T16:48:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.314+0000 7fde17200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde101568e0 0x7fde100c2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.314+0000 7fde17200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fde100c4d60 0x7fde100c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.314+0000 7fde17200640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde10151080 con 0x7fde10150ca0 2024-09-13T16:48:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.314+0000 7fde17200640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1015ae40 con 0x7fde100c4d60 2024-09-13T16:48:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.314+0000 7fde17200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1006a990 con 0x7fde101568e0 2024-09-13T16:48:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.314+0000 7fde07e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde101568e0 0x7fde100c2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.314+0000 7fde15200640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fde100c4d60 0x7fde100c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.314+0000 7fde07e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde101568e0 0x7fde100c2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:53714/0 (socket says 172.21.15.28:53714) 2024-09-13T16:48:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.314+0000 7fde07e00640 1 -- 172.21.15.28:0/4253749964 learned_addr learned my addr 172.21.15.28:0/4253749964 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.314+0000 7fde0fe00640 1 --1- 172.21.15.28:0/4253749964 >> v1:172.21.15.191:6789/0 conn(0x7fde10150ca0 0x7fde1015a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:41448/0 (socket says 172.21.15.28:41448) 2024-09-13T16:48:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.315+0000 7fde07e00640 1 -- 172.21.15.28:0/4253749964 >> v2:172.21.15.191:3300/0 conn(0x7fde100c4d60 msgr2=0x7fde100c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.315+0000 7fde07e00640 1 --2- 172.21.15.28:0/4253749964 >> v2:172.21.15.191:3300/0 conn(0x7fde100c4d60 0x7fde100c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.315+0000 7fde07e00640 1 -- 172.21.15.28:0/4253749964 >> v1:172.21.15.191:6789/0 conn(0x7fde10150ca0 legacy=0x7fde1015a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.315+0000 7fde07e00640 1 -- 172.21.15.28:0/4253749964 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde100c2870 con 0x7fde101568e0 2024-09-13T16:48:27.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.315+0000 7fde07e00640 1 --2- 172.21.15.28:0/4253749964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde101568e0 0x7fde100c2250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009870 tx=0x7fddf802a1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc97e6683ed4adf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:27.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.316+0000 7fde0f400640 1 -- 172.21.15.28:0/4253749964 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8032440 con 0x7fde101568e0 2024-09-13T16:48:27.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.316+0000 7fde0f400640 1 -- 172.21.15.28:0/4253749964 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf80325a0 con 0x7fde101568e0 2024-09-13T16:48:27.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.316+0000 7fde0f400640 1 -- 172.21.15.28:0/4253749964 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8032930 con 0x7fde101568e0 2024-09-13T16:48:27.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.316+0000 7fde17200640 1 -- 172.21.15.28:0/4253749964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde101568e0 msgr2=0x7fde100c2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.316+0000 7fde17200640 1 --2- 172.21.15.28:0/4253749964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde101568e0 0x7fde100c2250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009870 tx=0x7fddf802a1d0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.316+0000 7fde17200640 1 -- 172.21.15.28:0/4253749964 shutdown_connections 2024-09-13T16:48:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.316+0000 7fde17200640 1 --2- 172.21.15.28:0/4253749964 >> v2:172.21.15.191:3300/0 conn(0x7fde100c4d60 0x7fde100c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.316+0000 7fde17200640 1 --2- 172.21.15.28:0/4253749964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde101568e0 0x7fde100c2250 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.316+0000 7fde17200640 1 -- 172.21.15.28:0/4253749964 >> 172.21.15.28:0/4253749964 conn(0x7fde10066040 msgr2=0x7fde10069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.316+0000 7fde17200640 1 -- 172.21.15.28:0/4253749964 shutdown_connections 2024-09-13T16:48:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.317+0000 7fde17200640 1 -- 172.21.15.28:0/4253749964 wait complete. 2024-09-13T16:48:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.318+0000 7fde17200640 1 Processor -- start 2024-09-13T16:48:27.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.318+0000 7fde17200640 1 -- start start 2024-09-13T16:48:27.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.319+0000 7fde17200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde100c4d60 0x7fde10158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.319+0000 7fde17200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde10150ca0 0x7fde101dfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.319+0000 7fde17200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde101568e0 0x7fde101e0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.319+0000 7fde17200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1014b3a0 con 0x7fde10150ca0 2024-09-13T16:48:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.319+0000 7fde17200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde101e1b40 con 0x7fde101568e0 2024-09-13T16:48:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.319+0000 7fde17200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde101e1e00 con 0x7fde100c4d60 2024-09-13T16:48:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.319+0000 7fde07e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde10150ca0 0x7fde101dfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.319+0000 7fde07e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde10150ca0 0x7fde101dfcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39264/0 (socket says 172.21.15.28:39264) 2024-09-13T16:48:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.319+0000 7fde07e00640 1 -- 172.21.15.28:0/2138133161 learned_addr learned my addr 172.21.15.28:0/2138133161 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.319+0000 7fde0fe00640 1 --2- 172.21.15.28:0/2138133161 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde100c4d60 0x7fde10158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.319+0000 7fde15200640 1 --2- 172.21.15.28:0/2138133161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde101568e0 0x7fde101e0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.320+0000 7fde07e00640 1 -- 172.21.15.28:0/2138133161 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde100c4d60 msgr2=0x7fde10158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.320+0000 7fde07e00640 1 --2- 172.21.15.28:0/2138133161 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde100c4d60 0x7fde10158d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.320+0000 7fde07e00640 1 -- 172.21.15.28:0/2138133161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde101568e0 msgr2=0x7fde101e0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.320+0000 7fde07e00640 1 --2- 172.21.15.28:0/2138133161 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde101568e0 0x7fde101e0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.320+0000 7fde07e00640 1 -- 172.21.15.28:0/2138133161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf80094d0 con 0x7fde10150ca0 2024-09-13T16:48:27.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.320+0000 7fde07e00640 1 --2- 172.21.15.28:0/2138133161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde10150ca0 0x7fde101dfcd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fddf8000c00 tx=0x7fddf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:27.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.320+0000 7fde0cc00640 1 -- 172.21.15.28:0/2138133161 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8032440 con 0x7fde10150ca0 2024-09-13T16:48:27.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.320+0000 7fde0cc00640 1 -- 172.21.15.28:0/2138133161 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf8030980 con 0x7fde10150ca0 2024-09-13T16:48:27.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.320+0000 7fde0cc00640 1 -- 172.21.15.28:0/2138133161 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8030d30 con 0x7fde10150ca0 2024-09-13T16:48:27.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.320+0000 7fde17200640 1 -- 172.21.15.28:0/2138133161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde101ecf00 con 0x7fde10150ca0 2024-09-13T16:48:27.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.320+0000 7fde17200640 1 -- 172.21.15.28:0/2138133161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde10159c90 con 0x7fde10150ca0 2024-09-13T16:48:27.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.321+0000 7fde17200640 1 -- 172.21.15.28:0/2138133161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fde10159f30 con 0x7fde10150ca0 2024-09-13T16:48:27.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.322+0000 7fde0cc00640 1 -- 172.21.15.28:0/2138133161 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fddf8039040 con 0x7fde10150ca0 2024-09-13T16:48:27.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.323+0000 7fde0cc00640 1 --2- 172.21.15.28:0/2138133161 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fddec06ac00 0x7fddec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.323+0000 7fde0cc00640 1 -- 172.21.15.28:0/2138133161 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fddf80a9cf0 con 0x7fde10150ca0 2024-09-13T16:48:27.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.323+0000 7fde0cc00640 1 --2- 172.21.15.28:0/2138133161 >> [v2:172.21.15.28:6824/2616036674,v1:172.21.15.28:6825/2616036674] conn(0x7fddec072f20 0x7fddec0753a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.323+0000 7fde0cc00640 1 -- 172.21.15.28:0/2138133161 --> [v2:172.21.15.28:6824/2616036674,v1:172.21.15.28:6825/2616036674] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fddec075a70 con 0x7fddec072f20 2024-09-13T16:48:27.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.323+0000 7fde0cc00640 1 -- 172.21.15.28:0/2138133161 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddf80ac250 con 0x7fde10150ca0 2024-09-13T16:48:27.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.323+0000 7fde0fe00640 1 --2- 172.21.15.28:0/2138133161 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fddec06ac00 0x7fddec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.324+0000 7fde15200640 1 --2- 172.21.15.28:0/2138133161 >> [v2:172.21.15.28:6824/2616036674,v1:172.21.15.28:6825/2616036674] conn(0x7fddec072f20 0x7fddec0753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.324+0000 7fde0fe00640 1 --2- 172.21.15.28:0/2138133161 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fddec06ac00 0x7fddec06d0c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fde00000f60 tx=0x7fde00000ed0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:27.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.324+0000 7fde15200640 1 --2- 172.21.15.28:0/2138133161 >> [v2:172.21.15.28:6824/2616036674,v1:172.21.15.28:6825/2616036674] conn(0x7fddec072f20 0x7fddec0753a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:27.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.325+0000 7fde0cc00640 1 -- 172.21.15.28:0/2138133161 <== osd.3 v2:172.21.15.28:6824/2616036674 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fddec075a70 con 0x7fddec072f20 2024-09-13T16:48:27.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.338+0000 7fde17200640 1 -- 172.21.15.28:0/2138133161 --> [v2:172.21.15.28:6824/2616036674,v1:172.21.15.28:6825/2616036674] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fde100c2e30 con 0x7fddec072f20 2024-09-13T16:48:27.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.338+0000 7fde0cc00640 1 -- 172.21.15.28:0/2138133161 <== osd.3 v2:172.21.15.28:6824/2616036674 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fde0800f040 con 0x7fddec072f20 2024-09-13T16:48:27.339 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:48:27.340 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T16:48:27.340 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:48:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.338+0000 7fde17200640 1 -- 172.21.15.28:0/2138133161 >> [v2:172.21.15.28:6824/2616036674,v1:172.21.15.28:6825/2616036674] conn(0x7fddec072f20 msgr2=0x7fddec0753a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.338+0000 7fde17200640 1 --2- 172.21.15.28:0/2138133161 >> [v2:172.21.15.28:6824/2616036674,v1:172.21.15.28:6825/2616036674] conn(0x7fddec072f20 0x7fddec0753a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.338+0000 7fde17200640 1 -- 172.21.15.28:0/2138133161 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fddec06ac00 msgr2=0x7fddec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.338+0000 7fde17200640 1 --2- 172.21.15.28:0/2138133161 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fddec06ac00 0x7fddec06d0c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fde00000f60 tx=0x7fde00000ed0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.338+0000 7fde17200640 1 -- 172.21.15.28:0/2138133161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde10150ca0 msgr2=0x7fde101dfcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.338+0000 7fde17200640 1 --2- 172.21.15.28:0/2138133161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde10150ca0 0x7fde101dfcd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fddf8000c00 tx=0x7fddf80046c0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.338+0000 7fde15200640 1 -- 172.21.15.28:0/2138133161 reap_dead start 2024-09-13T16:48:27.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.339+0000 7fde17200640 1 -- 172.21.15.28:0/2138133161 shutdown_connections 2024-09-13T16:48:27.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.339+0000 7fde17200640 1 -- 172.21.15.28:0/2138133161 >> 172.21.15.28:0/2138133161 conn(0x7fde10066040 msgr2=0x7fde10156cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:27.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.339+0000 7fde17200640 1 -- 172.21.15.28:0/2138133161 shutdown_connections 2024-09-13T16:48:27.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.339+0000 7fde17200640 1 -- 172.21.15.28:0/2138133161 wait complete. 2024-09-13T16:48:27.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-13T16:48:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.511+0000 7fee6bc00640 1 Processor -- start 2024-09-13T16:48:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.511+0000 7fee6bc00640 1 -- start start 2024-09-13T16:48:27.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.512+0000 7fee6bc00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fee640c5140 0x7fee64159800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.512+0000 7fee6bc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee640c6f50 con 0x7fee640c7830 2024-09-13T16:48:27.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.512+0000 7fee6bc00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee640c6940 con 0x7fee640c73e0 2024-09-13T16:48:27.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.512+0000 7fee6bc00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee64159d40 con 0x7fee640c5140 2024-09-13T16:48:27.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.512+0000 7fee69c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fee640c7830 0x7fee6414f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52760/0 (socket says 172.21.15.28:52760) 2024-09-13T16:48:27.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.512+0000 7fee69c00640 1 -- 172.21.15.28:0/2359436182 learned_addr learned my addr 172.21.15.28:0/2359436182 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:27.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.512+0000 7fee69200640 1 --2- 172.21.15.28:0/2359436182 >> v2:172.21.15.191:3300/0 conn(0x7fee640c5140 0x7fee64159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.513+0000 7fee63400640 1 -- 172.21.15.28:0/2359436182 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077692289 0 0) 0x7fee640c6f50 con 0x7fee640c7830 2024-09-13T16:48:27.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.513+0000 7fee63400640 1 -- 172.21.15.28:0/2359436182 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee4c003520 con 0x7fee640c7830 2024-09-13T16:48:27.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.513+0000 7fee63400640 1 -- 172.21.15.28:0/2359436182 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2696385980 0 0) 0x7fee4c003520 con 0x7fee640c7830 2024-09-13T16:48:27.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.513+0000 7fee63400640 1 -- 172.21.15.28:0/2359436182 >> v2:172.21.15.191:3300/0 conn(0x7fee640c5140 msgr2=0x7fee64159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:27.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.513+0000 7fee63400640 1 --2- 172.21.15.28:0/2359436182 >> v2:172.21.15.191:3300/0 conn(0x7fee640c5140 0x7fee64159800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.513+0000 7fee63400640 1 -- 172.21.15.28:0/2359436182 >> v1:172.21.15.191:6789/0 conn(0x7fee640c73e0 legacy=0x7fee640c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.513+0000 7fee63400640 1 -- 172.21.15.28:0/2359436182 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee640c5c30 con 0x7fee640c7830 2024-09-13T16:48:27.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.513+0000 7fee63400640 1 -- 172.21.15.28:0/2359436182 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1654382301 0 0) 0x7fee5c0019f0 con 0x7fee640c7830 2024-09-13T16:48:27.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.513+0000 7fee63400640 1 -- 172.21.15.28:0/2359436182 >> v1:172.21.15.28:6789/0 conn(0x7fee640c7830 legacy=0x7fee6414f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.513+0000 7fee63400640 1 --2- 172.21.15.28:0/2359436182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee4c0039c0 0x7fee4c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.513+0000 7fee63400640 1 --2- 172.21.15.28:0/2359436182 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee4c0047c0 0x7fee4c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.513+0000 7fee63400640 1 --2- 172.21.15.28:0/2359436182 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee4c0051d0 0x7fee4c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.513+0000 7fee63400640 1 -- 172.21.15.28:0/2359436182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee64159d40 con 0x7fee4c0039c0 2024-09-13T16:48:27.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.513+0000 7fee63400640 1 -- 172.21.15.28:0/2359436182 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c00e330 con 0x7fee4c0047c0 2024-09-13T16:48:27.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.513+0000 7fee63400640 1 -- 172.21.15.28:0/2359436182 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c00e4f0 con 0x7fee4c0051d0 2024-09-13T16:48:27.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.513+0000 7fee69200640 1 --2- 172.21.15.28:0/2359436182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee4c0039c0 0x7fee4c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.513+0000 7fee63e00640 1 --2- 172.21.15.28:0/2359436182 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee4c0047c0 0x7fee4c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.514+0000 7fee69c00640 1 --2- 172.21.15.28:0/2359436182 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee4c0051d0 0x7fee4c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.514+0000 7fee69200640 1 -- 172.21.15.28:0/2359436182 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee4c0051d0 msgr2=0x7fee4c00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:27.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.514+0000 7fee69200640 1 --2- 172.21.15.28:0/2359436182 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee4c0051d0 0x7fee4c00db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.514+0000 7fee69200640 1 -- 172.21.15.28:0/2359436182 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee4c0047c0 msgr2=0x7fee4c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:27.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.514+0000 7fee69200640 1 --2- 172.21.15.28:0/2359436182 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee4c0047c0 0x7fee4c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.514+0000 7fee69200640 1 -- 172.21.15.28:0/2359436182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c003520 con 0x7fee4c0039c0 2024-09-13T16:48:27.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.514+0000 7fee69c00640 1 --2- 172.21.15.28:0/2359436182 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee4c0051d0 0x7fee4c00db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:48:27.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.514+0000 7fee63e00640 1 --2- 172.21.15.28:0/2359436182 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee4c0047c0 0x7fee4c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:27.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.514+0000 7fee69c00640 1 -- 172.21.15.28:0/2359436182 reap_dead start 2024-09-13T16:48:27.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.515+0000 7fee69200640 1 --2- 172.21.15.28:0/2359436182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee4c0039c0 0x7fee4c003de0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fee58009a90 tx=0x7fee4c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a586237f7cf33855 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:27.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.515+0000 7fee63400640 1 -- 172.21.15.28:0/2359436182 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58031400 con 0x7fee4c0039c0 2024-09-13T16:48:27.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.515+0000 7fee63400640 1 -- 172.21.15.28:0/2359436182 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5802d050 con 0x7fee4c0039c0 2024-09-13T16:48:27.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.515+0000 7fee63400640 1 -- 172.21.15.28:0/2359436182 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58031780 con 0x7fee4c0039c0 2024-09-13T16:48:27.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.516+0000 7fee6bc00640 1 -- 172.21.15.28:0/2359436182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee4c0039c0 msgr2=0x7fee4c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.516+0000 7fee6bc00640 1 --2- 172.21.15.28:0/2359436182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee4c0039c0 0x7fee4c003de0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fee58009a90 tx=0x7fee4c0057f0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.516+0000 7fee6bc00640 1 -- 172.21.15.28:0/2359436182 shutdown_connections 2024-09-13T16:48:27.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.516+0000 7fee6bc00640 1 --2- 172.21.15.28:0/2359436182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee4c0039c0 0x7fee4c003de0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.516+0000 7fee6bc00640 1 -- 172.21.15.28:0/2359436182 >> 172.21.15.28:0/2359436182 conn(0x7fee640ca6b0 msgr2=0x7fee640caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:27.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.516+0000 7fee6bc00640 1 -- 172.21.15.28:0/2359436182 shutdown_connections 2024-09-13T16:48:27.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.516+0000 7fee6bc00640 1 -- 172.21.15.28:0/2359436182 wait complete. 2024-09-13T16:48:27.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.517+0000 7fee6bc00640 1 Processor -- start 2024-09-13T16:48:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.518+0000 7fee6bc00640 1 -- start start 2024-09-13T16:48:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.518+0000 7fee6bc00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee4c0039c0 0x7fee640c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.518+0000 7fee6bc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee641563e0 0x7fee6413c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.519+0000 7fee6bc00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee6413c190 0x7fee64139b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.519+0000 7fee6bc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee641ee120 con 0x7fee641563e0 2024-09-13T16:48:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.519+0000 7fee6bc00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee641edc90 con 0x7fee4c0039c0 2024-09-13T16:48:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.519+0000 7fee6bc00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee641ed380 con 0x7fee6413c190 2024-09-13T16:48:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.519+0000 7fee63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee641563e0 0x7fee6413c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.519+0000 7fee63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee641563e0 0x7fee6413c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39276/0 (socket says 172.21.15.28:39276) 2024-09-13T16:48:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.519+0000 7fee63e00640 1 -- 172.21.15.28:0/2873016762 learned_addr learned my addr 172.21.15.28:0/2873016762 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.519+0000 7fee69200640 1 --2- 172.21.15.28:0/2873016762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee4c0039c0 0x7fee640c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.519+0000 7fee69c00640 1 --2- 172.21.15.28:0/2873016762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee6413c190 0x7fee64139b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.519+0000 7fee63e00640 1 -- 172.21.15.28:0/2873016762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee6413c190 msgr2=0x7fee64139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.519+0000 7fee63e00640 1 --2- 172.21.15.28:0/2873016762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee6413c190 0x7fee64139b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.519+0000 7fee63e00640 1 -- 172.21.15.28:0/2873016762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee4c0039c0 msgr2=0x7fee640c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.519+0000 7fee63e00640 1 --2- 172.21.15.28:0/2873016762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee4c0039c0 0x7fee640c97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.519+0000 7fee63e00640 1 -- 172.21.15.28:0/2873016762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee641409a0 con 0x7fee641563e0 2024-09-13T16:48:27.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.520+0000 7fee63e00640 1 --2- 172.21.15.28:0/2873016762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee641563e0 0x7fee6413c890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fee5000f990 tx=0x7fee5000fd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:27.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.520+0000 7fee60c00640 1 -- 172.21.15.28:0/2873016762 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee500052e0 con 0x7fee641563e0 2024-09-13T16:48:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.520+0000 7fee60c00640 1 -- 172.21.15.28:0/2873016762 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee50005440 con 0x7fee641563e0 2024-09-13T16:48:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.520+0000 7fee60c00640 1 -- 172.21.15.28:0/2873016762 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5000d910 con 0x7fee641563e0 2024-09-13T16:48:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.520+0000 7fee6bc00640 1 -- 172.21.15.28:0/2873016762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee64149110 con 0x7fee641563e0 2024-09-13T16:48:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.520+0000 7fee6bc00640 1 -- 172.21.15.28:0/2873016762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee641ed920 con 0x7fee641563e0 2024-09-13T16:48:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.521+0000 7fee6bc00640 1 -- 172.21.15.28:0/2873016762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee641edae0 con 0x7fee641563e0 2024-09-13T16:48:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.522+0000 7fee60c00640 1 -- 172.21.15.28:0/2873016762 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fee5000da70 con 0x7fee641563e0 2024-09-13T16:48:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.523+0000 7fee60c00640 1 --2- 172.21.15.28:0/2873016762 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fee4406ac30 0x7fee4406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.523+0000 7fee60c00640 1 -- 172.21.15.28:0/2873016762 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fee50089400 con 0x7fee641563e0 2024-09-13T16:48:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.523+0000 7fee69200640 1 --2- 172.21.15.28:0/2873016762 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fee4406ac30 0x7fee4406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.523+0000 7fee69200640 1 --2- 172.21.15.28:0/2873016762 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fee4406ac30 0x7fee4406d0f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fee58033000 tx=0x7fee58008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.527+0000 7fee60c00640 1 -- 172.21.15.28:0/2873016762 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fee500544c0 con 0x7fee641563e0 2024-09-13T16:48:27.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.705+0000 7fee6bc00640 1 -- 172.21.15.28:0/2873016762 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fee640c6be0 con 0x7fee4406ac30 2024-09-13T16:48:27.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.706+0000 7fee60c00640 1 -- 172.21.15.28:0/2873016762 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fee580041c0 con 0x7fee4406ac30 2024-09-13T16:48:27.708 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 3 to deep-scrub 2024-09-13T16:48:27.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.708+0000 7fee6bc00640 1 -- 172.21.15.28:0/2873016762 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fee4406ac30 msgr2=0x7fee4406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.708+0000 7fee6bc00640 1 --2- 172.21.15.28:0/2873016762 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fee4406ac30 0x7fee4406d0f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fee58033000 tx=0x7fee58008030 comp rx=0 tx=0).stop 2024-09-13T16:48:27.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.708+0000 7fee6bc00640 1 -- 172.21.15.28:0/2873016762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee641563e0 msgr2=0x7fee6413c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.708+0000 7fee6bc00640 1 --2- 172.21.15.28:0/2873016762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee641563e0 0x7fee6413c890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fee5000f990 tx=0x7fee5000fd90 comp rx=0 tx=0).stop 2024-09-13T16:48:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.709+0000 7fee6bc00640 1 -- 172.21.15.28:0/2873016762 shutdown_connections 2024-09-13T16:48:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.709+0000 7fee6bc00640 1 --2- 172.21.15.28:0/2873016762 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fee4406ac30 0x7fee4406d0f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.709+0000 7fee6bc00640 1 --2- 172.21.15.28:0/2873016762 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee6413c190 0x7fee64139b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.709+0000 7fee6bc00640 1 --2- 172.21.15.28:0/2873016762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee641563e0 0x7fee6413c890 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.709+0000 7fee6bc00640 1 --2- 172.21.15.28:0/2873016762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee4c0039c0 0x7fee640c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.709+0000 7fee6bc00640 1 -- 172.21.15.28:0/2873016762 >> 172.21.15.28:0/2873016762 conn(0x7fee640ca6b0 msgr2=0x7fee6406d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.709+0000 7fee6bc00640 1 -- 172.21.15.28:0/2873016762 shutdown_connections 2024-09-13T16:48:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.709+0000 7fee6bc00640 1 -- 172.21.15.28:0/2873016762 wait complete. 2024-09-13T16:48:27.727 INFO:tasks.ceph:Scrubbing osd.4 2024-09-13T16:48:27.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-13T16:48:27.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.885+0000 7f0270e00640 1 Processor -- start 2024-09-13T16:48:27.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.885+0000 7f0270e00640 1 -- start start 2024-09-13T16:48:27.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.886+0000 7f0270e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f026c150ca0 0x7f026c1568e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.886+0000 7f0270e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f026c0c2250 0x7f026c15a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.886+0000 7f0270e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f026c0c4d60 0x7f026c0c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.886+0000 7f0270e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f026c0c2670 con 0x7f026c0c2250 2024-09-13T16:48:27.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.886+0000 7f0270e00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f026c151080 con 0x7f026c150ca0 2024-09-13T16:48:27.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.886+0000 7f0270e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026c06a990 con 0x7f026c0c4d60 2024-09-13T16:48:27.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.887+0000 7f0269e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f026c150ca0 0x7f026c1568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.887+0000 7f026a800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f026c0c4d60 0x7f026c0c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.887+0000 7f0269e00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f026c150ca0 0x7f026c1568e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:38830/0 (socket says 172.21.15.28:38830) 2024-09-13T16:48:27.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.887+0000 7f0269e00640 1 -- 172.21.15.28:0/3396834940 learned_addr learned my addr 172.21.15.28:0/3396834940 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:27.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.887+0000 7f0261400640 1 --2- 172.21.15.28:0/3396834940 >> v2:172.21.15.28:3300/0 conn(0x7f026c0c2250 0x7f026c15a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.887+0000 7f026a800640 1 -- 172.21.15.28:0/3396834940 >> v2:172.21.15.191:3300/0 conn(0x7f026c150ca0 msgr2=0x7f026c1568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:27.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.887+0000 7f026a800640 1 --2- 172.21.15.28:0/3396834940 >> v2:172.21.15.191:3300/0 conn(0x7f026c150ca0 0x7f026c1568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.887+0000 7f026a800640 1 -- 172.21.15.28:0/3396834940 >> v2:172.21.15.28:3300/0 conn(0x7f026c0c2250 msgr2=0x7f026c15a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:27.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.887+0000 7f026a800640 1 --2- 172.21.15.28:0/3396834940 >> v2:172.21.15.28:3300/0 conn(0x7f026c0c2250 0x7f026c15a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.887+0000 7f026a800640 1 -- 172.21.15.28:0/3396834940 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c0c2870 con 0x7f026c0c4d60 2024-09-13T16:48:27.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.888+0000 7f026a800640 1 --2- 172.21.15.28:0/3396834940 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f026c0c4d60 0x7f026c0c51d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f026c072100 tx=0x7f026402a540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49e93933b5b135cc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:27.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.888+0000 7f0269400640 1 -- 172.21.15.28:0/3396834940 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264007740 con 0x7f026c0c4d60 2024-09-13T16:48:27.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.888+0000 7f0269400640 1 -- 172.21.15.28:0/3396834940 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026402d050 con 0x7f026c0c4d60 2024-09-13T16:48:27.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.888+0000 7f0269400640 1 -- 172.21.15.28:0/3396834940 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264007ac0 con 0x7f026c0c4d60 2024-09-13T16:48:27.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.889+0000 7f0270e00640 1 -- 172.21.15.28:0/3396834940 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f026c0c4d60 msgr2=0x7f026c0c51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.889+0000 7f0270e00640 1 --2- 172.21.15.28:0/3396834940 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f026c0c4d60 0x7f026c0c51d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f026c072100 tx=0x7f026402a540 comp rx=0 tx=0).stop 2024-09-13T16:48:27.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.889+0000 7f0270e00640 1 -- 172.21.15.28:0/3396834940 shutdown_connections 2024-09-13T16:48:27.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.889+0000 7f0270e00640 1 --2- 172.21.15.28:0/3396834940 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f026c0c4d60 0x7f026c0c51d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.889+0000 7f0270e00640 1 --2- 172.21.15.28:0/3396834940 >> v2:172.21.15.28:3300/0 conn(0x7f026c0c2250 0x7f026c15a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.889+0000 7f0270e00640 1 --2- 172.21.15.28:0/3396834940 >> v2:172.21.15.191:3300/0 conn(0x7f026c150ca0 0x7f026c1568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.889+0000 7f0270e00640 1 -- 172.21.15.28:0/3396834940 >> 172.21.15.28:0/3396834940 conn(0x7f026c066040 msgr2=0x7f026c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:27.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.889+0000 7f0270e00640 1 -- 172.21.15.28:0/3396834940 shutdown_connections 2024-09-13T16:48:27.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.889+0000 7f0270e00640 1 -- 172.21.15.28:0/3396834940 wait complete. 2024-09-13T16:48:27.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.891+0000 7f0270e00640 1 Processor -- start 2024-09-13T16:48:27.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.891+0000 7f0270e00640 1 -- start start 2024-09-13T16:48:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.892+0000 7f0270e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f026c0c2250 0x7f026c1ec530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.892+0000 7f0270e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f026c0c4d60 0x7f026c1eca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.892+0000 7f0270e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f026c150ca0 0x7f026c158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.892+0000 7f0270e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c1474d0 con 0x7f026c0c2250 2024-09-13T16:48:27.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.892+0000 7f0270e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c0bf6a0 con 0x7f026c150ca0 2024-09-13T16:48:27.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.892+0000 7f0270e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026c0bfbd0 con 0x7f026c0c4d60 2024-09-13T16:48:27.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.892+0000 7f0269e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f026c0c2250 0x7f026c1ec530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.892+0000 7f0269e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f026c0c2250 0x7f026c1ec530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39298/0 (socket says 172.21.15.28:39298) 2024-09-13T16:48:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.892+0000 7f0269e00640 1 -- 172.21.15.28:0/2180190676 learned_addr learned my addr 172.21.15.28:0/2180190676 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.892+0000 7f026a800640 1 --2- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f026c150ca0 0x7f026c158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.892+0000 7f0261400640 1 --2- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f026c0c4d60 0x7f026c1eca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.892+0000 7f0269e00640 1 -- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f026c0c4d60 msgr2=0x7f026c1eca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.892+0000 7f0269e00640 1 --2- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f026c0c4d60 0x7f026c1eca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.893+0000 7f0269e00640 1 -- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f026c150ca0 msgr2=0x7f026c158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.893+0000 7f0269e00640 1 --2- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f026c150ca0 0x7f026c158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.893+0000 7f0269e00640 1 -- 172.21.15.28:0/2180190676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0264009a70 con 0x7f026c0c2250 2024-09-13T16:48:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.893+0000 7f026a800640 1 --2- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f026c150ca0 0x7f026c158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:48:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.893+0000 7f0261400640 1 --2- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f026c0c4d60 0x7f026c1eca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:48:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.893+0000 7f0269e00640 1 --2- 172.21.15.28:0/2180190676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f026c0c2250 0x7f026c1ec530 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f025c00e840 tx=0x7f025c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:27.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.893+0000 7f0262a00640 1 -- 172.21.15.28:0/2180190676 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c00c8e0 con 0x7f026c0c2250 2024-09-13T16:48:27.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.893+0000 7f0262a00640 1 -- 172.21.15.28:0/2180190676 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025c00ca40 con 0x7f026c0c2250 2024-09-13T16:48:27.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.893+0000 7f0262a00640 1 -- 172.21.15.28:0/2180190676 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c00cdf0 con 0x7f026c0c2250 2024-09-13T16:48:27.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.893+0000 7f0270e00640 1 -- 172.21.15.28:0/2180190676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f026c147720 con 0x7f026c0c2250 2024-09-13T16:48:27.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.893+0000 7f0270e00640 1 -- 172.21.15.28:0/2180190676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f026c0be400 con 0x7f026c0c2250 2024-09-13T16:48:27.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.894+0000 7f0270e00640 1 -- 172.21.15.28:0/2180190676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f026c159f30 con 0x7f026c0c2250 2024-09-13T16:48:27.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.895+0000 7f0262a00640 1 -- 172.21.15.28:0/2180190676 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f025c013470 con 0x7f026c0c2250 2024-09-13T16:48:27.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.896+0000 7f0262a00640 1 --2- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f024806ac00 0x7f024806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.896+0000 7f0262a00640 1 -- 172.21.15.28:0/2180190676 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f025c089910 con 0x7f026c0c2250 2024-09-13T16:48:27.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.896+0000 7f0262a00640 1 --2- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:6800/2817102450,v1:172.21.15.191:6801/2817102450] conn(0x7f0248072f20 0x7f02480753a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:27.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.896+0000 7f0262a00640 1 -- 172.21.15.28:0/2180190676 --> [v2:172.21.15.191:6800/2817102450,v1:172.21.15.191:6801/2817102450] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0248075a70 con 0x7f0248072f20 2024-09-13T16:48:27.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.896+0000 7f0262a00640 1 -- 172.21.15.28:0/2180190676 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f025c08e2a0 con 0x7f026c0c2250 2024-09-13T16:48:27.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.896+0000 7f0261400640 1 --2- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f024806ac00 0x7f024806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.897+0000 7f026a800640 1 --2- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:6800/2817102450,v1:172.21.15.191:6801/2817102450] conn(0x7f0248072f20 0x7f02480753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:27.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.897+0000 7f0261400640 1 --2- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f024806ac00 0x7f024806d0c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f026c0c5220 tx=0x7f0254008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:27.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.897+0000 7f026a800640 1 --2- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:6800/2817102450,v1:172.21.15.191:6801/2817102450] conn(0x7f0248072f20 0x7f02480753a0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:27.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.898+0000 7f0262a00640 1 -- 172.21.15.28:0/2180190676 <== osd.4 v2:172.21.15.191:6800/2817102450 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f0248075a70 con 0x7f0248072f20 2024-09-13T16:48:27.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.911+0000 7f0270e00640 1 -- 172.21.15.28:0/2180190676 --> [v2:172.21.15.191:6800/2817102450,v1:172.21.15.191:6801/2817102450] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f026c0c2e30 con 0x7f0248072f20 2024-09-13T16:48:27.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.911+0000 7f0262a00640 1 -- 172.21.15.28:0/2180190676 <== osd.4 v2:172.21.15.191:6800/2817102450 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f0264007870 con 0x7f0248072f20 2024-09-13T16:48:27.913 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:48:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T16:48:27.914 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:48:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.912+0000 7f0270e00640 1 -- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:6800/2817102450,v1:172.21.15.191:6801/2817102450] conn(0x7f0248072f20 msgr2=0x7f02480753a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.912+0000 7f0270e00640 1 --2- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:6800/2817102450,v1:172.21.15.191:6801/2817102450] conn(0x7f0248072f20 0x7f02480753a0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.912+0000 7f0270e00640 1 -- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f024806ac00 msgr2=0x7f024806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.912+0000 7f0270e00640 1 --2- 172.21.15.28:0/2180190676 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f024806ac00 0x7f024806d0c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f026c0c5220 tx=0x7f0254008040 comp rx=0 tx=0).stop 2024-09-13T16:48:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.912+0000 7f0270e00640 1 -- 172.21.15.28:0/2180190676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f026c0c2250 msgr2=0x7f026c1ec530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.912+0000 7f0270e00640 1 --2- 172.21.15.28:0/2180190676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f026c0c2250 0x7f026c1ec530 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f025c00e840 tx=0x7f025c00ed00 comp rx=0 tx=0).stop 2024-09-13T16:48:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.912+0000 7f026a800640 1 -- 172.21.15.28:0/2180190676 reap_dead start 2024-09-13T16:48:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.912+0000 7f0270e00640 1 -- 172.21.15.28:0/2180190676 shutdown_connections 2024-09-13T16:48:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.912+0000 7f0270e00640 1 -- 172.21.15.28:0/2180190676 >> 172.21.15.28:0/2180190676 conn(0x7f026c066040 msgr2=0x7f026c0c5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.912+0000 7f0270e00640 1 -- 172.21.15.28:0/2180190676 shutdown_connections 2024-09-13T16:48:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:27.912+0000 7f0270e00640 1 -- 172.21.15.28:0/2180190676 wait complete. 2024-09-13T16:48:27.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-13T16:48:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.083+0000 7fdcd5c00640 1 Processor -- start 2024-09-13T16:48:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.083+0000 7fdcd5c00640 1 -- start start 2024-09-13T16:48:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.084+0000 7fdcd5c00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdcd00c5140 0x7fdcd0159800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.084+0000 7fdcd5c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd00c6f50 con 0x7fdcd00c7830 2024-09-13T16:48:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.084+0000 7fdcd5c00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd00c6940 con 0x7fdcd00c73e0 2024-09-13T16:48:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.084+0000 7fdcd5c00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd0159d40 con 0x7fdcd00c5140 2024-09-13T16:48:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.084+0000 7fdccf400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdcd00c7830 0x7fdcd014f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52772/0 (socket says 172.21.15.28:52772) 2024-09-13T16:48:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.084+0000 7fdccf400640 1 -- 172.21.15.28:0/1657667090 learned_addr learned my addr 172.21.15.28:0/1657667090 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.084+0000 7fdccea00640 1 --2- 172.21.15.28:0/1657667090 >> v2:172.21.15.191:3300/0 conn(0x7fdcd00c5140 0x7fdcd0159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.085+0000 7fdccea00640 1 -- 172.21.15.28:0/1657667090 >> v1:172.21.15.191:6789/0 conn(0x7fdcd00c73e0 legacy=0x7fdcd00c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.085+0000 7fdccea00640 1 -- 172.21.15.28:0/1657667090 >> v1:172.21.15.28:6789/0 conn(0x7fdcd00c7830 legacy=0x7fdcd014f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.085+0000 7fdccea00640 1 -- 172.21.15.28:0/1657667090 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd00c5c30 con 0x7fdcd00c5140 2024-09-13T16:48:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.085+0000 7fdccd600640 1 -- 172.21.15.28:0/1657667090 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989074050 0 0) 0x7fdcd00c6f50 con 0x7fdcd00c7830 2024-09-13T16:48:28.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.086+0000 7fdccea00640 1 --2- 172.21.15.28:0/1657667090 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdcd00c5140 0x7fdcd0159800 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00097a0 tx=0x7fdcc002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9681ffbb077302fb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:28.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.086+0000 7fdccd600640 1 -- 172.21.15.28:0/1657667090 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0032440 con 0x7fdcd00c5140 2024-09-13T16:48:28.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.086+0000 7fdccd600640 1 -- 172.21.15.28:0/1657667090 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc00325a0 con 0x7fdcd00c5140 2024-09-13T16:48:28.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.086+0000 7fdccd600640 1 -- 172.21.15.28:0/1657667090 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0032930 con 0x7fdcd00c5140 2024-09-13T16:48:28.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.086+0000 7fdcd5c00640 1 -- 172.21.15.28:0/1657667090 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdcd00c5140 msgr2=0x7fdcd0159800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.086+0000 7fdcd5c00640 1 --2- 172.21.15.28:0/1657667090 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdcd00c5140 0x7fdcd0159800 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00097a0 tx=0x7fdcc002a1a0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.087+0000 7fdcd5c00640 1 -- 172.21.15.28:0/1657667090 shutdown_connections 2024-09-13T16:48:28.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.087+0000 7fdcd5c00640 1 --2- 172.21.15.28:0/1657667090 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdcd00c5140 0x7fdcd0159800 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.087+0000 7fdcd5c00640 1 -- 172.21.15.28:0/1657667090 >> 172.21.15.28:0/1657667090 conn(0x7fdcd00ca6b0 msgr2=0x7fdcd00caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:28.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.087+0000 7fdcd5c00640 1 -- 172.21.15.28:0/1657667090 shutdown_connections 2024-09-13T16:48:28.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.087+0000 7fdcd5c00640 1 -- 172.21.15.28:0/1657667090 wait complete. 2024-09-13T16:48:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.088+0000 7fdcd5c00640 1 Processor -- start 2024-09-13T16:48:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.088+0000 7fdcd5c00640 1 -- start start 2024-09-13T16:48:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.089+0000 7fdcd5c00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdcd00c5140 0x7fdcd01ef610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.089+0000 7fdcd5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcd00c73e0 0x7fdcd0147b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.089+0000 7fdcd5c00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdcd0159940 0x7fdcd01ee670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.089+0000 7fdcd5c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd01ed2a0 con 0x7fdcd00c73e0 2024-09-13T16:48:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.089+0000 7fdcd5c00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd01f0360 con 0x7fdcd00c5140 2024-09-13T16:48:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.089+0000 7fdcd5c00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd01f0080 con 0x7fdcd0159940 2024-09-13T16:48:28.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.089+0000 7fdcce000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcd00c73e0 0x7fdcd0147b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.090+0000 7fdcce000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcd00c73e0 0x7fdcd0147b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39306/0 (socket says 172.21.15.28:39306) 2024-09-13T16:48:28.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.090+0000 7fdcce000640 1 -- 172.21.15.28:0/2930750774 learned_addr learned my addr 172.21.15.28:0/2930750774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:28.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.090+0000 7fdccea00640 1 --2- 172.21.15.28:0/2930750774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdcd00c5140 0x7fdcd01ef610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.090+0000 7fdccf400640 1 --2- 172.21.15.28:0/2930750774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdcd0159940 0x7fdcd01ee670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.090+0000 7fdcce000640 1 -- 172.21.15.28:0/2930750774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdcd0159940 msgr2=0x7fdcd01ee670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.090+0000 7fdcce000640 1 --2- 172.21.15.28:0/2930750774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdcd0159940 0x7fdcd01ee670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.090+0000 7fdcce000640 1 -- 172.21.15.28:0/2930750774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdcd00c5140 msgr2=0x7fdcd01ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.090+0000 7fdcce000640 1 --2- 172.21.15.28:0/2930750774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdcd00c5140 0x7fdcd01ef610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.090+0000 7fdcce000640 1 -- 172.21.15.28:0/2930750774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc0009400 con 0x7fdcd00c73e0 2024-09-13T16:48:28.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.090+0000 7fdcce000640 1 --2- 172.21.15.28:0/2930750774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcd00c73e0 0x7fdcd0147b10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc00c450 tx=0x7fdcbc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:28.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.090+0000 7fdcc6a00640 1 -- 172.21.15.28:0/2930750774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc0052d0 con 0x7fdcd00c73e0 2024-09-13T16:48:28.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.091+0000 7fdcc6a00640 1 -- 172.21.15.28:0/2930750774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcbc005430 con 0x7fdcd00c73e0 2024-09-13T16:48:28.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.091+0000 7fdcc6a00640 1 -- 172.21.15.28:0/2930750774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc012870 con 0x7fdcd00c73e0 2024-09-13T16:48:28.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.091+0000 7fdcd5c00640 1 -- 172.21.15.28:0/2930750774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd0147030 con 0x7fdcd00c73e0 2024-09-13T16:48:28.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.091+0000 7fdcd5c00640 1 -- 172.21.15.28:0/2930750774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd01475f0 con 0x7fdcd00c73e0 2024-09-13T16:48:28.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.092+0000 7fdcd5c00640 1 -- 172.21.15.28:0/2930750774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd0146e90 con 0x7fdcd00c73e0 2024-09-13T16:48:28.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.092+0000 7fdcc6a00640 1 -- 172.21.15.28:0/2930750774 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdcbc0129d0 con 0x7fdcd00c73e0 2024-09-13T16:48:28.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.093+0000 7fdcc6a00640 1 --2- 172.21.15.28:0/2930750774 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fdcb006ac30 0x7fdcb006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.093+0000 7fdcc6a00640 1 -- 172.21.15.28:0/2930750774 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdcbc088e30 con 0x7fdcd00c73e0 2024-09-13T16:48:28.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.094+0000 7fdccea00640 1 --2- 172.21.15.28:0/2930750774 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fdcb006ac30 0x7fdcb006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.094+0000 7fdccea00640 1 --2- 172.21.15.28:0/2930750774 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fdcb006ac30 0x7fdcb006d0f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0034000 tx=0x7fdcc002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:28.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.097+0000 7fdcc6a00640 1 -- 172.21.15.28:0/2930750774 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fdcd0146e90 con 0x7fdcd00c73e0 2024-09-13T16:48:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.277+0000 7fdcd5c00640 1 -- 172.21.15.28:0/2930750774 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fdcd00c6be0 con 0x7fdcb006ac30 2024-09-13T16:48:28.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.277+0000 7fdcc6a00640 1 -- 172.21.15.28:0/2930750774 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdcc0032a60 con 0x7fdcb006ac30 2024-09-13T16:48:28.279 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 4 to deep-scrub 2024-09-13T16:48:28.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.280+0000 7fdcd5c00640 1 -- 172.21.15.28:0/2930750774 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fdcb006ac30 msgr2=0x7fdcb006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.280+0000 7fdcd5c00640 1 --2- 172.21.15.28:0/2930750774 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fdcb006ac30 0x7fdcb006d0f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0034000 tx=0x7fdcc002e000 comp rx=0 tx=0).stop 2024-09-13T16:48:28.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.280+0000 7fdcd5c00640 1 -- 172.21.15.28:0/2930750774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcd00c73e0 msgr2=0x7fdcd0147b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.280+0000 7fdcd5c00640 1 --2- 172.21.15.28:0/2930750774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcd00c73e0 0x7fdcd0147b10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc00c450 tx=0x7fdcbc00c910 comp rx=0 tx=0).stop 2024-09-13T16:48:28.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.280+0000 7fdcd5c00640 1 -- 172.21.15.28:0/2930750774 shutdown_connections 2024-09-13T16:48:28.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.280+0000 7fdcd5c00640 1 --2- 172.21.15.28:0/2930750774 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fdcb006ac30 0x7fdcb006d0f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.280+0000 7fdcd5c00640 1 --2- 172.21.15.28:0/2930750774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdcd0159940 0x7fdcd01ee670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.280+0000 7fdcd5c00640 1 --2- 172.21.15.28:0/2930750774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcd00c73e0 0x7fdcd0147b10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.280+0000 7fdcd5c00640 1 --2- 172.21.15.28:0/2930750774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdcd00c5140 0x7fdcd01ef610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.280+0000 7fdcd5c00640 1 -- 172.21.15.28:0/2930750774 >> 172.21.15.28:0/2930750774 conn(0x7fdcd00ca6b0 msgr2=0x7fdcd00caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:28.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.280+0000 7fdcd5c00640 1 -- 172.21.15.28:0/2930750774 shutdown_connections 2024-09-13T16:48:28.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.280+0000 7fdcd5c00640 1 -- 172.21.15.28:0/2930750774 wait complete. 2024-09-13T16:48:28.297 INFO:tasks.ceph:Scrubbing osd.5 2024-09-13T16:48:28.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-13T16:48:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.459+0000 7fa38be00640 1 Processor -- start 2024-09-13T16:48:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.459+0000 7fa38be00640 1 -- start start 2024-09-13T16:48:28.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.460+0000 7fa38be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa38c152c00 0x7fa38c0c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.460+0000 7fa38be00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa38c0c3430 0x7fa38c0c9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.460+0000 7fa38be00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38c152fe0 con 0x7fa38c0c95d0 2024-09-13T16:48:28.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.460+0000 7fa38be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa38c159820 con 0x7fa38c152c00 2024-09-13T16:48:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.460+0000 7fa38be00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa38c1589c0 con 0x7fa38c0c3430 2024-09-13T16:48:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.460+0000 7fa38aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa38c152c00 0x7fa38c0c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.460+0000 7fa38aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa38c152c00 0x7fa38c0c9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39310/0 (socket says 172.21.15.28:39310) 2024-09-13T16:48:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.460+0000 7fa38aa00640 1 -- 172.21.15.28:0/126649711 learned_addr learned my addr 172.21.15.28:0/126649711 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.460+0000 7fa383e00640 1 --2- 172.21.15.28:0/126649711 >> v2:172.21.15.191:3300/0 conn(0x7fa38c0c3430 0x7fa38c0c9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.461+0000 7fa38aa00640 1 -- 172.21.15.28:0/126649711 >> v2:172.21.15.191:3300/0 conn(0x7fa38c0c3430 msgr2=0x7fa38c0c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.461+0000 7fa38aa00640 1 --2- 172.21.15.28:0/126649711 >> v2:172.21.15.191:3300/0 conn(0x7fa38c0c3430 0x7fa38c0c9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.461+0000 7fa38aa00640 1 -- 172.21.15.28:0/126649711 >> v1:172.21.15.191:6789/0 conn(0x7fa38c0c95d0 legacy=0x7fa38c06acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.461+0000 7fa38aa00640 1 -- 172.21.15.28:0/126649711 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38c158ca0 con 0x7fa38c152c00 2024-09-13T16:48:28.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.462+0000 7fa38aa00640 1 --2- 172.21.15.28:0/126649711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa38c152c00 0x7fa38c0c9e10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa38c0cb8c0 tx=0x7fa37c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0d539f8da88edf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:28.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.462+0000 7fa38a000640 1 -- 172.21.15.28:0/126649711 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37c032440 con 0x7fa38c152c00 2024-09-13T16:48:28.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.462+0000 7fa38a000640 1 -- 172.21.15.28:0/126649711 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37c0325a0 con 0x7fa38c152c00 2024-09-13T16:48:28.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.462+0000 7fa38a000640 1 -- 172.21.15.28:0/126649711 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37c032930 con 0x7fa38c152c00 2024-09-13T16:48:28.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.463+0000 7fa38be00640 1 -- 172.21.15.28:0/126649711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa38c152c00 msgr2=0x7fa38c0c9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.463+0000 7fa38be00640 1 --2- 172.21.15.28:0/126649711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa38c152c00 0x7fa38c0c9e10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa38c0cb8c0 tx=0x7fa37c02a270 comp rx=0 tx=0).stop 2024-09-13T16:48:28.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.463+0000 7fa38be00640 1 -- 172.21.15.28:0/126649711 shutdown_connections 2024-09-13T16:48:28.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.463+0000 7fa38be00640 1 --2- 172.21.15.28:0/126649711 >> v2:172.21.15.191:3300/0 conn(0x7fa38c0c3430 0x7fa38c0c9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.463+0000 7fa38be00640 1 --2- 172.21.15.28:0/126649711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa38c152c00 0x7fa38c0c9e10 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.463+0000 7fa38be00640 1 -- 172.21.15.28:0/126649711 >> 172.21.15.28:0/126649711 conn(0x7fa38c065c50 msgr2=0x7fa38c066030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.463+0000 7fa38be00640 1 -- 172.21.15.28:0/126649711 shutdown_connections 2024-09-13T16:48:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.463+0000 7fa38be00640 1 -- 172.21.15.28:0/126649711 wait complete. 2024-09-13T16:48:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.466+0000 7fa38be00640 1 Processor -- start 2024-09-13T16:48:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.466+0000 7fa38be00640 1 -- start start 2024-09-13T16:48:28.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.467+0000 7fa38be00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa38c0c3430 0x7fa38c156720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.467+0000 7fa38be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa38c0c95d0 0x7fa38c0b3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.467+0000 7fa38be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa38c152c00 0x7fa38c148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.467+0000 7fa38be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c1ef280 con 0x7fa38c0c95d0 2024-09-13T16:48:28.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.467+0000 7fa38be00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c1ef7b0 con 0x7fa38c152c00 2024-09-13T16:48:28.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.467+0000 7fa38be00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c1580a0 con 0x7fa38c0c3430 2024-09-13T16:48:28.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.467+0000 7fa383e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa38c0c95d0 0x7fa38c0b3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.467+0000 7fa38aa00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa38c0c3430 0x7fa38c156720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.467+0000 7fa383e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa38c0c95d0 0x7fa38c0b3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39320/0 (socket says 172.21.15.28:39320) 2024-09-13T16:48:28.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.467+0000 7fa38b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa38c152c00 0x7fa38c148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.467+0000 7fa383e00640 1 -- 172.21.15.28:0/4049327269 learned_addr learned my addr 172.21.15.28:0/4049327269 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:28.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.468+0000 7fa383e00640 1 -- 172.21.15.28:0/4049327269 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa38c0c3430 msgr2=0x7fa38c156720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.468+0000 7fa383e00640 1 --2- 172.21.15.28:0/4049327269 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa38c0c3430 0x7fa38c156720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.468+0000 7fa383e00640 1 -- 172.21.15.28:0/4049327269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa38c152c00 msgr2=0x7fa38c148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.468+0000 7fa383e00640 1 --2- 172.21.15.28:0/4049327269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa38c152c00 0x7fa38c148e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.468+0000 7fa383e00640 1 -- 172.21.15.28:0/4049327269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c0094d0 con 0x7fa38c0c95d0 2024-09-13T16:48:28.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.468+0000 7fa383e00640 1 --2- 172.21.15.28:0/4049327269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa38c0c95d0 0x7fa38c0b3680 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa37400d800 tx=0x7fa37400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:28.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.469+0000 7fa382a00640 1 -- 172.21.15.28:0/4049327269 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37400b6c0 con 0x7fa38c0c95d0 2024-09-13T16:48:28.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.469+0000 7fa382a00640 1 -- 172.21.15.28:0/4049327269 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37400b820 con 0x7fa38c0c95d0 2024-09-13T16:48:28.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.469+0000 7fa382a00640 1 -- 172.21.15.28:0/4049327269 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37400bbd0 con 0x7fa38c0c95d0 2024-09-13T16:48:28.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.469+0000 7fa38be00640 1 -- 172.21.15.28:0/4049327269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa38c0c8500 con 0x7fa38c0c95d0 2024-09-13T16:48:28.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.469+0000 7fa38be00640 1 -- 172.21.15.28:0/4049327269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa38c1ef4d0 con 0x7fa38c0c95d0 2024-09-13T16:48:28.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.470+0000 7fa38be00640 1 -- 172.21.15.28:0/4049327269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa38c159540 con 0x7fa38c0c95d0 2024-09-13T16:48:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.471+0000 7fa382a00640 1 -- 172.21.15.28:0/4049327269 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa37400bd30 con 0x7fa38c0c95d0 2024-09-13T16:48:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.472+0000 7fa382a00640 1 --2- 172.21.15.28:0/4049327269 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa36806ac00 0x7fa36806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.472+0000 7fa382a00640 1 -- 172.21.15.28:0/4049327269 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa3740898c0 con 0x7fa38c0c95d0 2024-09-13T16:48:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.472+0000 7fa382a00640 1 --2- 172.21.15.28:0/4049327269 >> [v2:172.21.15.191:6816/2503027971,v1:172.21.15.191:6817/2503027971] conn(0x7fa368072f20 0x7fa3680753a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.472+0000 7fa382a00640 1 -- 172.21.15.28:0/4049327269 --> [v2:172.21.15.191:6816/2503027971,v1:172.21.15.191:6817/2503027971] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa368075a70 con 0x7fa368072f20 2024-09-13T16:48:28.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.472+0000 7fa38aa00640 1 --2- 172.21.15.28:0/4049327269 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa36806ac00 0x7fa36806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.472+0000 7fa382a00640 1 -- 172.21.15.28:0/4049327269 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa37408e2a0 con 0x7fa38c0c95d0 2024-09-13T16:48:28.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.473+0000 7fa38b400640 1 --2- 172.21.15.28:0/4049327269 >> [v2:172.21.15.191:6816/2503027971,v1:172.21.15.191:6817/2503027971] conn(0x7fa368072f20 0x7fa3680753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.473+0000 7fa38aa00640 1 --2- 172.21.15.28:0/4049327269 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa36806ac00 0x7fa36806d0c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa37c034000 tx=0x7fa37c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:28.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.473+0000 7fa38b400640 1 --2- 172.21.15.28:0/4049327269 >> [v2:172.21.15.191:6816/2503027971,v1:172.21.15.191:6817/2503027971] conn(0x7fa368072f20 0x7fa3680753a0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:28.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.474+0000 7fa382a00640 1 -- 172.21.15.28:0/4049327269 <== osd.5 v2:172.21.15.191:6816/2503027971 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fa368075a70 con 0x7fa368072f20 2024-09-13T16:48:28.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.487+0000 7fa38be00640 1 -- 172.21.15.28:0/4049327269 --> [v2:172.21.15.191:6816/2503027971,v1:172.21.15.191:6817/2503027971] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa38c159260 con 0x7fa368072f20 2024-09-13T16:48:28.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.487+0000 7fa382a00640 1 -- 172.21.15.28:0/4049327269 <== osd.5 v2:172.21.15.191:6816/2503027971 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fa384001b20 con 0x7fa368072f20 2024-09-13T16:48:28.489 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:48:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T16:48:28.489 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:48:28.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.487+0000 7fa38be00640 1 -- 172.21.15.28:0/4049327269 >> [v2:172.21.15.191:6816/2503027971,v1:172.21.15.191:6817/2503027971] conn(0x7fa368072f20 msgr2=0x7fa3680753a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.487+0000 7fa38be00640 1 --2- 172.21.15.28:0/4049327269 >> [v2:172.21.15.191:6816/2503027971,v1:172.21.15.191:6817/2503027971] conn(0x7fa368072f20 0x7fa3680753a0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.487+0000 7fa38be00640 1 -- 172.21.15.28:0/4049327269 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa36806ac00 msgr2=0x7fa36806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.487+0000 7fa38be00640 1 --2- 172.21.15.28:0/4049327269 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa36806ac00 0x7fa36806d0c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa37c034000 tx=0x7fa37c02e000 comp rx=0 tx=0).stop 2024-09-13T16:48:28.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.487+0000 7fa38be00640 1 -- 172.21.15.28:0/4049327269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa38c0c95d0 msgr2=0x7fa38c0b3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.487+0000 7fa38be00640 1 --2- 172.21.15.28:0/4049327269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa38c0c95d0 0x7fa38c0b3680 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa37400d800 tx=0x7fa37400dcc0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.488+0000 7fa38b400640 1 -- 172.21.15.28:0/4049327269 reap_dead start 2024-09-13T16:48:28.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.488+0000 7fa38be00640 1 -- 172.21.15.28:0/4049327269 shutdown_connections 2024-09-13T16:48:28.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.488+0000 7fa38be00640 1 -- 172.21.15.28:0/4049327269 >> 172.21.15.28:0/4049327269 conn(0x7fa38c065c50 msgr2=0x7fa38c0c4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:28.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.488+0000 7fa38be00640 1 -- 172.21.15.28:0/4049327269 shutdown_connections 2024-09-13T16:48:28.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.488+0000 7fa38be00640 1 -- 172.21.15.28:0/4049327269 wait complete. 2024-09-13T16:48:28.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-13T16:48:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.660+0000 7f18b8800640 1 Processor -- start 2024-09-13T16:48:28.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.660+0000 7f18b8800640 1 -- start start 2024-09-13T16:48:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.661+0000 7f18b8800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f18b0152d50 0x7f18b00c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.661+0000 7f18b8800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b00c5770 0x7f18b006fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.661+0000 7f18b8800640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b00c6cf0 con 0x7f18b00c5340 2024-09-13T16:48:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.661+0000 7f18b8800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b015c080 con 0x7f18b0152d50 2024-09-13T16:48:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.661+0000 7f18b8800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b015cbc0 con 0x7f18b00c5770 2024-09-13T16:48:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.662+0000 7f18b5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f18b0152d50 0x7f18b00c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.662+0000 7f18b5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f18b0152d50 0x7f18b00c67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39336/0 (socket says 172.21.15.28:39336) 2024-09-13T16:48:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.662+0000 7f18b5e00640 1 -- 172.21.15.28:0/3344247969 learned_addr learned my addr 172.21.15.28:0/3344247969 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.662+0000 7f18b5400640 1 --1- 172.21.15.28:0/3344247969 >> v1:172.21.15.191:6789/0 conn(0x7f18b00c5340 0x7f18b006d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:41484/0 (socket says 172.21.15.28:41484) 2024-09-13T16:48:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.662+0000 7f18b6800640 1 --2- 172.21.15.28:0/3344247969 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b00c5770 0x7f18b006fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.662+0000 7f18b5e00640 1 -- 172.21.15.28:0/3344247969 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b00c5770 msgr2=0x7f18b006fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.662+0000 7f18b5e00640 1 --2- 172.21.15.28:0/3344247969 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b00c5770 0x7f18b006fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.662+0000 7f18b5e00640 1 -- 172.21.15.28:0/3344247969 >> v1:172.21.15.191:6789/0 conn(0x7f18b00c5340 legacy=0x7f18b006d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.662+0000 7f18b5e00640 1 -- 172.21.15.28:0/3344247969 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b00c5d10 con 0x7f18b0152d50 2024-09-13T16:48:28.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.662+0000 7f18b4a00640 1 -- 172.21.15.28:0/3344247969 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724521646 0 0) 0x7f18a0001110 con 0x7f18b00c5340 2024-09-13T16:48:28.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.663+0000 7f18b5e00640 1 --2- 172.21.15.28:0/3344247969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b0152d50 0x7f18b00c67b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f18a4009870 tx=0x7f18a402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28583c9cd0ae8a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:28.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.663+0000 7f18b4a00640 1 -- 172.21.15.28:0/3344247969 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a4032640 con 0x7f18b0152d50 2024-09-13T16:48:28.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.663+0000 7f18b4a00640 1 -- 172.21.15.28:0/3344247969 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a40327a0 con 0x7f18b0152d50 2024-09-13T16:48:28.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.663+0000 7f18b4a00640 1 -- 172.21.15.28:0/3344247969 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a4032b30 con 0x7f18b0152d50 2024-09-13T16:48:28.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.664+0000 7f18b8800640 1 -- 172.21.15.28:0/3344247969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b0152d50 msgr2=0x7f18b00c67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.664+0000 7f18b8800640 1 --2- 172.21.15.28:0/3344247969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b0152d50 0x7f18b00c67b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f18a4009870 tx=0x7f18a402a300 comp rx=0 tx=0).stop 2024-09-13T16:48:28.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.664+0000 7f18b8800640 1 -- 172.21.15.28:0/3344247969 shutdown_connections 2024-09-13T16:48:28.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.664+0000 7f18b8800640 1 --2- 172.21.15.28:0/3344247969 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b00c5770 0x7f18b006fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.664+0000 7f18b8800640 1 --2- 172.21.15.28:0/3344247969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b0152d50 0x7f18b00c67b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.664+0000 7f18b8800640 1 -- 172.21.15.28:0/3344247969 >> 172.21.15.28:0/3344247969 conn(0x7f18b00ca6b0 msgr2=0x7f18b00caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:28.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.664+0000 7f18b8800640 1 -- 172.21.15.28:0/3344247969 shutdown_connections 2024-09-13T16:48:28.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.664+0000 7f18b8800640 1 -- 172.21.15.28:0/3344247969 wait complete. 2024-09-13T16:48:28.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.666+0000 7f18b8800640 1 Processor -- start 2024-09-13T16:48:28.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.666+0000 7f18b8800640 1 -- start start 2024-09-13T16:48:28.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.666+0000 7f18b8800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b00c5340 0x7f18b01eecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.666+0000 7f18b8800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18b00c5770 0x7f18b0154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.667+0000 7f18b8800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b0152d50 0x7f18b0159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.667+0000 7f18b8800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b01ef210 con 0x7f18b0152d50 2024-09-13T16:48:28.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.667+0000 7f18b8800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b014d090 con 0x7f18b00c5770 2024-09-13T16:48:28.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.667+0000 7f18b8800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b01556e0 con 0x7f18b00c5340 2024-09-13T16:48:28.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.667+0000 7f18b6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b0152d50 0x7f18b0159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.667+0000 7f18b6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b0152d50 0x7f18b0159eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39342/0 (socket says 172.21.15.28:39342) 2024-09-13T16:48:28.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.667+0000 7f18b6800640 1 -- 172.21.15.28:0/2204704078 learned_addr learned my addr 172.21.15.28:0/2204704078 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:28.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.667+0000 7f18b5400640 1 --2- 172.21.15.28:0/2204704078 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18b00c5770 0x7f18b0154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.667+0000 7f18b5e00640 1 --2- 172.21.15.28:0/2204704078 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b00c5340 0x7f18b01eecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.667+0000 7f18b6800640 1 -- 172.21.15.28:0/2204704078 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b00c5340 msgr2=0x7f18b01eecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.667+0000 7f18b6800640 1 --2- 172.21.15.28:0/2204704078 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b00c5340 0x7f18b01eecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.667+0000 7f18b6800640 1 -- 172.21.15.28:0/2204704078 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18b00c5770 msgr2=0x7f18b0154e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.667+0000 7f18b6800640 1 --2- 172.21.15.28:0/2204704078 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18b00c5770 0x7f18b0154e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.667+0000 7f18b6800640 1 -- 172.21.15.28:0/2204704078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a40094d0 con 0x7f18b0152d50 2024-09-13T16:48:28.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.668+0000 7f18b6800640 1 --2- 172.21.15.28:0/2204704078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b0152d50 0x7f18b0159eb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00efc0 tx=0x7f18ac00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:28.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.668+0000 7f18aa000640 1 -- 172.21.15.28:0/2204704078 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac014480 con 0x7f18b0152d50 2024-09-13T16:48:28.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.668+0000 7f18aa000640 1 -- 172.21.15.28:0/2204704078 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ac00f040 con 0x7f18b0152d50 2024-09-13T16:48:28.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.668+0000 7f18aa000640 1 -- 172.21.15.28:0/2204704078 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ac014820 con 0x7f18b0152d50 2024-09-13T16:48:28.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.668+0000 7f18b8800640 1 -- 172.21.15.28:0/2204704078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b0155980 con 0x7f18b0152d50 2024-09-13T16:48:28.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.668+0000 7f18b8800640 1 -- 172.21.15.28:0/2204704078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b014d5e0 con 0x7f18b0152d50 2024-09-13T16:48:28.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.669+0000 7f18b8800640 1 -- 172.21.15.28:0/2204704078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b00c67b0 con 0x7f18b0152d50 2024-09-13T16:48:28.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.670+0000 7f18aa000640 1 -- 172.21.15.28:0/2204704078 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f18ac014980 con 0x7f18b0152d50 2024-09-13T16:48:28.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.671+0000 7f18aa000640 1 --2- 172.21.15.28:0/2204704078 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f189406ac00 0x7f189406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:28.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.671+0000 7f18aa000640 1 -- 172.21.15.28:0/2204704078 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f18ac089ed0 con 0x7f18b0152d50 2024-09-13T16:48:28.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.671+0000 7f18b5e00640 1 --2- 172.21.15.28:0/2204704078 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f189406ac00 0x7f189406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:28.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.672+0000 7f18b5e00640 1 --2- 172.21.15.28:0/2204704078 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f189406ac00 0x7f189406d0c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f18a4034000 tx=0x7f18a402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:28.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.675+0000 7f18aa000640 1 -- 172.21.15.28:0/2204704078 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f18ac055040 con 0x7f18b0152d50 2024-09-13T16:48:28.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.853+0000 7f18b8800640 1 -- 172.21.15.28:0/2204704078 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f18b0153c80 con 0x7f189406ac00 2024-09-13T16:48:28.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.854+0000 7f18aa000640 1 -- 172.21.15.28:0/2204704078 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f18b0153c80 con 0x7f189406ac00 2024-09-13T16:48:28.855 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 5 to deep-scrub 2024-09-13T16:48:28.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.856+0000 7f18b8800640 1 -- 172.21.15.28:0/2204704078 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f189406ac00 msgr2=0x7f189406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.856+0000 7f18b8800640 1 --2- 172.21.15.28:0/2204704078 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f189406ac00 0x7f189406d0c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f18a4034000 tx=0x7f18a402e000 comp rx=0 tx=0).stop 2024-09-13T16:48:28.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.856+0000 7f18b8800640 1 -- 172.21.15.28:0/2204704078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b0152d50 msgr2=0x7f18b0159eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:28.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.856+0000 7f18b8800640 1 --2- 172.21.15.28:0/2204704078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b0152d50 0x7f18b0159eb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00efc0 tx=0x7f18ac00c490 comp rx=0 tx=0).stop 2024-09-13T16:48:28.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.856+0000 7f18b8800640 1 -- 172.21.15.28:0/2204704078 shutdown_connections 2024-09-13T16:48:28.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.856+0000 7f18b8800640 1 --2- 172.21.15.28:0/2204704078 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f189406ac00 0x7f189406d0c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.856+0000 7f18b8800640 1 --2- 172.21.15.28:0/2204704078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b0152d50 0x7f18b0159eb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.856+0000 7f18b8800640 1 --2- 172.21.15.28:0/2204704078 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18b00c5770 0x7f18b0154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.856+0000 7f18b8800640 1 --2- 172.21.15.28:0/2204704078 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f18b00c5340 0x7f18b01eecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.856+0000 7f18b8800640 1 -- 172.21.15.28:0/2204704078 >> 172.21.15.28:0/2204704078 conn(0x7f18b00ca6b0 msgr2=0x7f18b01e2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.856+0000 7f18b8800640 1 -- 172.21.15.28:0/2204704078 shutdown_connections 2024-09-13T16:48:28.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:28.856+0000 7f18b8800640 1 -- 172.21.15.28:0/2204704078 wait complete. 2024-09-13T16:48:28.873 INFO:tasks.ceph:Scrubbing osd.6 2024-09-13T16:48:28.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-13T16:48:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.032+0000 7fe7ab200640 1 Processor -- start 2024-09-13T16:48:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.032+0000 7fe7ab200640 1 -- start start 2024-09-13T16:48:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.033+0000 7fe7ab200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7a4159760 0x7fe7a4069e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.033+0000 7fe7ab200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a41592d0 con 0x7fe7a406ad40 2024-09-13T16:48:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.033+0000 7fe7ab200640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a406b120 con 0x7fe7a406b5e0 2024-09-13T16:48:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.033+0000 7fe7ab200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a4070680 con 0x7fe7a4159760 2024-09-13T16:48:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.033+0000 7fe7a3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe7a406ad40 0x7fe7a4070b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50302/0 (socket says 172.21.15.28:50302) 2024-09-13T16:48:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.033+0000 7fe7a3e00640 1 -- 172.21.15.28:0/3911655635 learned_addr learned my addr 172.21.15.28:0/3911655635 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.033+0000 7fe7a9200640 1 --2- 172.21.15.28:0/3911655635 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7a4159760 0x7fe7a4069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.034+0000 7fe7a3400640 1 -- 172.21.15.28:0/3911655635 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876366828 0 0) 0x7fe7a41592d0 con 0x7fe7a406ad40 2024-09-13T16:48:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.034+0000 7fe7a3400640 1 -- 172.21.15.28:0/3911655635 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe788003520 con 0x7fe7a406ad40 2024-09-13T16:48:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.034+0000 7fe7a3400640 1 -- 172.21.15.28:0/3911655635 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698862747 0 0) 0x7fe7a406b120 con 0x7fe7a406b5e0 2024-09-13T16:48:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.034+0000 7fe7a3400640 1 -- 172.21.15.28:0/3911655635 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7a41592d0 con 0x7fe7a406b5e0 2024-09-13T16:48:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.034+0000 7fe7a9200640 1 -- 172.21.15.28:0/3911655635 >> v1:172.21.15.191:6789/0 conn(0x7fe7a406b5e0 legacy=0x7fe7a4071760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.034+0000 7fe7a9200640 1 -- 172.21.15.28:0/3911655635 >> v1:172.21.15.28:6789/0 conn(0x7fe7a406ad40 legacy=0x7fe7a4070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.034+0000 7fe7a9200640 1 -- 172.21.15.28:0/3911655635 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a4070890 con 0x7fe7a4159760 2024-09-13T16:48:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.035+0000 7fe7a9200640 1 --2- 172.21.15.28:0/3911655635 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7a4159760 0x7fe7a4069e80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe79c009dc0 tx=0x7fe79c02a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce4ac015e1ea2c74 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.035+0000 7fe7a3400640 1 -- 172.21.15.28:0/3911655635 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79c007740 con 0x7fe7a4159760 2024-09-13T16:48:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.035+0000 7fe7a3400640 1 -- 172.21.15.28:0/3911655635 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79c02d050 con 0x7fe7a4159760 2024-09-13T16:48:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.035+0000 7fe7a3400640 1 -- 172.21.15.28:0/3911655635 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79c007ac0 con 0x7fe7a4159760 2024-09-13T16:48:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.036+0000 7fe7ab200640 1 -- 172.21.15.28:0/3911655635 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7a4159760 msgr2=0x7fe7a4069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.036+0000 7fe7ab200640 1 --2- 172.21.15.28:0/3911655635 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7a4159760 0x7fe7a4069e80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe79c009dc0 tx=0x7fe79c02a760 comp rx=0 tx=0).stop 2024-09-13T16:48:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.036+0000 7fe7ab200640 1 -- 172.21.15.28:0/3911655635 shutdown_connections 2024-09-13T16:48:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.036+0000 7fe7ab200640 1 --2- 172.21.15.28:0/3911655635 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7a4159760 0x7fe7a4069e80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.036+0000 7fe7ab200640 1 -- 172.21.15.28:0/3911655635 >> 172.21.15.28:0/3911655635 conn(0x7fe7a40c7ad0 msgr2=0x7fe7a40c7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.036+0000 7fe7ab200640 1 -- 172.21.15.28:0/3911655635 shutdown_connections 2024-09-13T16:48:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.036+0000 7fe7ab200640 1 -- 172.21.15.28:0/3911655635 wait complete. 2024-09-13T16:48:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.038+0000 7fe7ab200640 1 Processor -- start 2024-09-13T16:48:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.038+0000 7fe7ab200640 1 -- start start 2024-09-13T16:48:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe7ab200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7a406ad40 0x7fe7a40ad3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe7ab200640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7a406b5e0 0x7fe7a414afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe7ab200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7a4159760 0x7fe7a40ae910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe7ab200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a40ad8e0 con 0x7fe7a4159760 2024-09-13T16:48:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe7ab200640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a40ca670 con 0x7fe7a406ad40 2024-09-13T16:48:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe7ab200640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a41df040 con 0x7fe7a406b5e0 2024-09-13T16:48:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe7a3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7a406ad40 0x7fe7a40ad3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe7a9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7a4159760 0x7fe7a40ae910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe79b400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7a406b5e0 0x7fe7a414afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe7a3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7a406ad40 0x7fe7a40ad3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:46706/0 (socket says 172.21.15.28:46706) 2024-09-13T16:48:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe7a3e00640 1 -- 172.21.15.28:0/1650111058 learned_addr learned my addr 172.21.15.28:0/1650111058 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe79b400640 1 -- 172.21.15.28:0/1650111058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7a406ad40 msgr2=0x7fe7a40ad3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe79b400640 1 --2- 172.21.15.28:0/1650111058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7a406ad40 0x7fe7a40ad3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe79b400640 1 -- 172.21.15.28:0/1650111058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7a4159760 msgr2=0x7fe7a40ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe79b400640 1 --2- 172.21.15.28:0/1650111058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7a4159760 0x7fe7a40ae910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe79b400640 1 -- 172.21.15.28:0/1650111058 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79c009a70 con 0x7fe7a406b5e0 2024-09-13T16:48:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe7a3e00640 1 --2- 172.21.15.28:0/1650111058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7a406ad40 0x7fe7a40ad3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.039+0000 7fe7a9200640 1 --2- 172.21.15.28:0/1650111058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7a4159760 0x7fe7a40ae910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.040+0000 7fe79b400640 1 --2- 172.21.15.28:0/1650111058 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7a406b5e0 0x7fe7a414afe0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00c400 tx=0x7fe78c00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.040+0000 7fe7a0c00640 1 -- 172.21.15.28:0/1650111058 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c0052d0 con 0x7fe7a406b5e0 2024-09-13T16:48:29.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.040+0000 7fe7a0c00640 1 -- 172.21.15.28:0/1650111058 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78c005430 con 0x7fe7a406b5e0 2024-09-13T16:48:29.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.040+0000 7fe7a0c00640 1 -- 172.21.15.28:0/1650111058 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c003830 con 0x7fe7a406b5e0 2024-09-13T16:48:29.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.040+0000 7fe7ab200640 1 -- 172.21.15.28:0/1650111058 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a41dfaa0 con 0x7fe7a406b5e0 2024-09-13T16:48:29.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.040+0000 7fe7ab200640 1 -- 172.21.15.28:0/1650111058 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a4156760 con 0x7fe7a406b5e0 2024-09-13T16:48:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.041+0000 7fe7ab200640 1 -- 172.21.15.28:0/1650111058 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe7a4158f60 con 0x7fe7a406b5e0 2024-09-13T16:48:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.042+0000 7fe7a0c00640 1 -- 172.21.15.28:0/1650111058 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe78c003990 con 0x7fe7a406b5e0 2024-09-13T16:48:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.043+0000 7fe7a0c00640 1 --2- 172.21.15.28:0/1650111058 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe78006ac30 0x7fe78006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.043+0000 7fe7a0c00640 1 -- 172.21.15.28:0/1650111058 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe78c0890d0 con 0x7fe7a406b5e0 2024-09-13T16:48:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.043+0000 7fe7a3e00640 1 --2- 172.21.15.28:0/1650111058 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe78006ac30 0x7fe78006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.043+0000 7fe7a0c00640 1 --2- 172.21.15.28:0/1650111058 >> [v2:172.21.15.191:6808/3006595088,v1:172.21.15.191:6809/3006595088] conn(0x7fe780072f50 0x7fe7800753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.043+0000 7fe7a0c00640 1 -- 172.21.15.28:0/1650111058 --> [v2:172.21.15.191:6808/3006595088,v1:172.21.15.191:6809/3006595088] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe780075aa0 con 0x7fe780072f50 2024-09-13T16:48:29.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.043+0000 7fe7a0c00640 1 -- 172.21.15.28:0/1650111058 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe78c089440 con 0x7fe7a406b5e0 2024-09-13T16:48:29.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.044+0000 7fe7a9200640 1 --2- 172.21.15.28:0/1650111058 >> [v2:172.21.15.191:6808/3006595088,v1:172.21.15.191:6809/3006595088] conn(0x7fe780072f50 0x7fe7800753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.044+0000 7fe7a3e00640 1 --2- 172.21.15.28:0/1650111058 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe78006ac30 0x7fe78006d0f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe794000f60 tx=0x7fe794009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:29.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.045+0000 7fe7a9200640 1 --2- 172.21.15.28:0/1650111058 >> [v2:172.21.15.191:6808/3006595088,v1:172.21.15.191:6809/3006595088] conn(0x7fe780072f50 0x7fe7800753d0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:29.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.045+0000 7fe7a0c00640 1 -- 172.21.15.28:0/1650111058 <== osd.6 v2:172.21.15.191:6808/3006595088 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fe780075aa0 con 0x7fe780072f50 2024-09-13T16:48:29.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.058+0000 7fe7ab200640 1 -- 172.21.15.28:0/1650111058 --> [v2:172.21.15.191:6808/3006595088,v1:172.21.15.191:6809/3006595088] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe7a4069e80 con 0x7fe780072f50 2024-09-13T16:48:29.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.059+0000 7fe7a0c00640 1 -- 172.21.15.28:0/1650111058 <== osd.6 v2:172.21.15.191:6808/3006595088 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe7a4069e80 con 0x7fe780072f50 2024-09-13T16:48:29.061 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:48:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T16:48:29.061 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:48:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.059+0000 7fe7ab200640 1 -- 172.21.15.28:0/1650111058 >> [v2:172.21.15.191:6808/3006595088,v1:172.21.15.191:6809/3006595088] conn(0x7fe780072f50 msgr2=0x7fe7800753d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.059+0000 7fe7ab200640 1 --2- 172.21.15.28:0/1650111058 >> [v2:172.21.15.191:6808/3006595088,v1:172.21.15.191:6809/3006595088] conn(0x7fe780072f50 0x7fe7800753d0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.059+0000 7fe7ab200640 1 -- 172.21.15.28:0/1650111058 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe78006ac30 msgr2=0x7fe78006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.059+0000 7fe7ab200640 1 --2- 172.21.15.28:0/1650111058 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe78006ac30 0x7fe78006d0f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe794000f60 tx=0x7fe794009040 comp rx=0 tx=0).stop 2024-09-13T16:48:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.059+0000 7fe7ab200640 1 -- 172.21.15.28:0/1650111058 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7a406b5e0 msgr2=0x7fe7a414afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.059+0000 7fe7ab200640 1 --2- 172.21.15.28:0/1650111058 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7a406b5e0 0x7fe7a414afe0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe78c00c400 tx=0x7fe78c00c8c0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.059+0000 7fe7a9200640 1 -- 172.21.15.28:0/1650111058 reap_dead start 2024-09-13T16:48:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.059+0000 7fe7ab200640 1 -- 172.21.15.28:0/1650111058 shutdown_connections 2024-09-13T16:48:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.059+0000 7fe7ab200640 1 -- 172.21.15.28:0/1650111058 >> 172.21.15.28:0/1650111058 conn(0x7fe7a40c7ad0 msgr2=0x7fe7a4157180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.059+0000 7fe7ab200640 1 -- 172.21.15.28:0/1650111058 shutdown_connections 2024-09-13T16:48:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.059+0000 7fe7ab200640 1 -- 172.21.15.28:0/1650111058 wait complete. 2024-09-13T16:48:29.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-13T16:48:29.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.230+0000 7f381be00640 1 Processor -- start 2024-09-13T16:48:29.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.230+0000 7f381be00640 1 -- start start 2024-09-13T16:48:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.232+0000 7f381be00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f381c150990 0x7f381c0c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.232+0000 7f381be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f381c0c44c0 0x7f381c156840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.232+0000 7f381be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381c150d70 con 0x7f381c0c17f0 2024-09-13T16:48:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.232+0000 7f381be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f381c0c5310 con 0x7f381c0c44c0 2024-09-13T16:48:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.232+0000 7f381be00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381c0c2190 con 0x7f381c150990 2024-09-13T16:48:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.232+0000 7f381b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f381c0c44c0 0x7f381c156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.232+0000 7f381b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f381c0c44c0 0x7f381c156840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38184/0 (socket says 172.21.15.28:38184) 2024-09-13T16:48:29.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.232+0000 7f381b400640 1 -- 172.21.15.28:0/254516940 learned_addr learned my addr 172.21.15.28:0/254516940 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:29.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.232+0000 7f381aa00640 1 --2- 172.21.15.28:0/254516940 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f381c150990 0x7f381c0c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.232+0000 7f381b400640 1 -- 172.21.15.28:0/254516940 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f381c150990 msgr2=0x7f381c0c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:29.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.232+0000 7f381b400640 1 --2- 172.21.15.28:0/254516940 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f381c150990 0x7f381c0c4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.232+0000 7f381b400640 1 -- 172.21.15.28:0/254516940 >> v1:172.21.15.28:6789/0 conn(0x7f381c0c17f0 legacy=0x7f381c06b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.232+0000 7f381b400640 1 -- 172.21.15.28:0/254516940 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381c0c2470 con 0x7f381c0c44c0 2024-09-13T16:48:29.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.232+0000 7f381a000640 1 -- 172.21.15.28:0/254516940 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709239162 0 0) 0x7f381c150d70 con 0x7f381c0c17f0 2024-09-13T16:48:29.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.233+0000 7f381b400640 1 --2- 172.21.15.28:0/254516940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c44c0 0x7f381c156840 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3814009e50 tx=0x7f381402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e542b6c569566c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:29.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.234+0000 7f381a000640 1 -- 172.21.15.28:0/254516940 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3814031600 con 0x7f381c0c44c0 2024-09-13T16:48:29.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.234+0000 7f381a000640 1 -- 172.21.15.28:0/254516940 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f381402d050 con 0x7f381c0c44c0 2024-09-13T16:48:29.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.234+0000 7f381a000640 1 -- 172.21.15.28:0/254516940 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3814031980 con 0x7f381c0c44c0 2024-09-13T16:48:29.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.234+0000 7f381be00640 1 -- 172.21.15.28:0/254516940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c44c0 msgr2=0x7f381c156840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.234+0000 7f381be00640 1 --2- 172.21.15.28:0/254516940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c44c0 0x7f381c156840 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3814009e50 tx=0x7f381402a710 comp rx=0 tx=0).stop 2024-09-13T16:48:29.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.234+0000 7f381be00640 1 -- 172.21.15.28:0/254516940 shutdown_connections 2024-09-13T16:48:29.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.234+0000 7f381be00640 1 --2- 172.21.15.28:0/254516940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c44c0 0x7f381c156840 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.234+0000 7f381be00640 1 --2- 172.21.15.28:0/254516940 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f381c150990 0x7f381c0c4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.234+0000 7f381be00640 1 -- 172.21.15.28:0/254516940 >> 172.21.15.28:0/254516940 conn(0x7f381c0ca2f0 msgr2=0x7f381c0ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:29.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.234+0000 7f381be00640 1 -- 172.21.15.28:0/254516940 shutdown_connections 2024-09-13T16:48:29.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.234+0000 7f381be00640 1 -- 172.21.15.28:0/254516940 wait complete. 2024-09-13T16:48:29.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.236+0000 7f381be00640 1 Processor -- start 2024-09-13T16:48:29.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.236+0000 7f381be00640 1 -- start start 2024-09-13T16:48:29.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.237+0000 7f381be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c17f0 0x7f381c071880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.237+0000 7f381be00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f381c0c44c0 0x7f381c155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.237+0000 7f381be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f381c150990 0x7f381c1442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.237+0000 7f381be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c0c85d0 con 0x7f381c0c17f0 2024-09-13T16:48:29.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.237+0000 7f381be00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c0c82f0 con 0x7f381c150990 2024-09-13T16:48:29.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.237+0000 7f381be00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381c155180 con 0x7f381c0c44c0 2024-09-13T16:48:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.237+0000 7f381aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c17f0 0x7f381c071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.237+0000 7f3813e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f381c0c44c0 0x7f381c155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.237+0000 7f381b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f381c150990 0x7f381c1442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.237+0000 7f381aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c17f0 0x7f381c071880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38192/0 (socket says 172.21.15.28:38192) 2024-09-13T16:48:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.237+0000 7f381aa00640 1 -- 172.21.15.28:0/4073219687 learned_addr learned my addr 172.21.15.28:0/4073219687 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.237+0000 7f381b400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f381c150990 0x7f381c1442b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:46712/0 (socket says 172.21.15.28:46712) 2024-09-13T16:48:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.238+0000 7f381aa00640 1 -- 172.21.15.28:0/4073219687 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f381c0c44c0 msgr2=0x7f381c155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.238+0000 7f381aa00640 1 --2- 172.21.15.28:0/4073219687 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f381c0c44c0 0x7f381c155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.238+0000 7f381aa00640 1 -- 172.21.15.28:0/4073219687 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f381c150990 msgr2=0x7f381c1442b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.238+0000 7f381aa00640 1 --2- 172.21.15.28:0/4073219687 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f381c150990 0x7f381c1442b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.238+0000 7f381aa00640 1 -- 172.21.15.28:0/4073219687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3814009b30 con 0x7f381c0c17f0 2024-09-13T16:48:29.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.238+0000 7f381aa00640 1 --2- 172.21.15.28:0/4073219687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c17f0 0x7f381c071880 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f380c00e9b0 tx=0x7f380c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:29.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.238+0000 7f3812a00640 1 -- 172.21.15.28:0/4073219687 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c009670 con 0x7f381c0c17f0 2024-09-13T16:48:29.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.238+0000 7f3812a00640 1 -- 172.21.15.28:0/4073219687 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380c00c9a0 con 0x7f381c0c17f0 2024-09-13T16:48:29.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.238+0000 7f3812a00640 1 -- 172.21.15.28:0/4073219687 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c00cd50 con 0x7f381c0c17f0 2024-09-13T16:48:29.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.238+0000 7f381be00640 1 -- 172.21.15.28:0/4073219687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f381c155460 con 0x7f381c0c17f0 2024-09-13T16:48:29.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.238+0000 7f381be00640 1 -- 172.21.15.28:0/4073219687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f381c155a20 con 0x7f381c0c17f0 2024-09-13T16:48:29.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.239+0000 7f381be00640 1 -- 172.21.15.28:0/4073219687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f381c0c3ab0 con 0x7f381c0c17f0 2024-09-13T16:48:29.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.240+0000 7f3812a00640 1 -- 172.21.15.28:0/4073219687 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f380c013470 con 0x7f381c0c17f0 2024-09-13T16:48:29.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.241+0000 7f3812a00640 1 --2- 172.21.15.28:0/4073219687 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f37f806ac00 0x7f37f806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.241+0000 7f3812a00640 1 -- 172.21.15.28:0/4073219687 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f380c0895d0 con 0x7f381c0c17f0 2024-09-13T16:48:29.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.241+0000 7f3813e00640 1 --2- 172.21.15.28:0/4073219687 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f37f806ac00 0x7f37f806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.242+0000 7f3813e00640 1 --2- 172.21.15.28:0/4073219687 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f37f806ac00 0x7f37f806d0c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f381c1e2e60 tx=0x7f3804009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:29.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.245+0000 7f3812a00640 1 -- 172.21.15.28:0/4073219687 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f380c0547d0 con 0x7f381c0c17f0 2024-09-13T16:48:29.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.424+0000 7f381be00640 1 -- 172.21.15.28:0/4073219687 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f381c0c2d10 con 0x7f37f806ac00 2024-09-13T16:48:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.424+0000 7f3812a00640 1 -- 172.21.15.28:0/4073219687 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3804001420 con 0x7f37f806ac00 2024-09-13T16:48:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 6 to deep-scrub 2024-09-13T16:48:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.426+0000 7f381be00640 1 -- 172.21.15.28:0/4073219687 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f37f806ac00 msgr2=0x7f37f806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.426+0000 7f381be00640 1 --2- 172.21.15.28:0/4073219687 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f37f806ac00 0x7f37f806d0c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f381c1e2e60 tx=0x7f3804009040 comp rx=0 tx=0).stop 2024-09-13T16:48:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.426+0000 7f381be00640 1 -- 172.21.15.28:0/4073219687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c17f0 msgr2=0x7f381c071880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.426+0000 7f381be00640 1 --2- 172.21.15.28:0/4073219687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c17f0 0x7f381c071880 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f380c00e9b0 tx=0x7f380c00ee70 comp rx=0 tx=0).stop 2024-09-13T16:48:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.427+0000 7f381be00640 1 -- 172.21.15.28:0/4073219687 shutdown_connections 2024-09-13T16:48:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.427+0000 7f381be00640 1 --2- 172.21.15.28:0/4073219687 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f37f806ac00 0x7f37f806d0c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.427+0000 7f381be00640 1 --2- 172.21.15.28:0/4073219687 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f381c150990 0x7f381c1442b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.427+0000 7f381be00640 1 --2- 172.21.15.28:0/4073219687 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f381c0c44c0 0x7f381c155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.427+0000 7f381be00640 1 --2- 172.21.15.28:0/4073219687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c17f0 0x7f381c071880 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.427+0000 7f381be00640 1 -- 172.21.15.28:0/4073219687 >> 172.21.15.28:0/4073219687 conn(0x7f381c0ca2f0 msgr2=0x7f381c0c6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.427+0000 7f381be00640 1 -- 172.21.15.28:0/4073219687 shutdown_connections 2024-09-13T16:48:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.427+0000 7f381be00640 1 -- 172.21.15.28:0/4073219687 wait complete. 2024-09-13T16:48:29.444 INFO:tasks.ceph:Scrubbing osd.7 2024-09-13T16:48:29.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-13T16:48:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.603+0000 7f8522600640 1 Processor -- start 2024-09-13T16:48:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.603+0000 7f8522600640 1 -- start start 2024-09-13T16:48:29.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.604+0000 7f8522600640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f851c150600 0x7f851c0c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.605+0000 7f8522600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f851c15a3c0 0x7f851c0c4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.605+0000 7f8522600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851c06e5b0 con 0x7f851c06ac20 2024-09-13T16:48:29.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.605+0000 7f8522600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f851c0c5740 con 0x7f851c15a3c0 2024-09-13T16:48:29.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.605+0000 7f8522600640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f851c0c4df0 con 0x7f851c150600 2024-09-13T16:48:29.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.605+0000 7f851be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f851c06ac20 0x7f851c159cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50322/0 (socket says 172.21.15.28:50322) 2024-09-13T16:48:29.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.605+0000 7f851be00640 1 -- 172.21.15.28:0/2156833837 learned_addr learned my addr 172.21.15.28:0/2156833837 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:29.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.605+0000 7f8521c00640 1 --2- 172.21.15.28:0/2156833837 >> v2:172.21.15.28:3300/0 conn(0x7f851c15a3c0 0x7f851c0c4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.605+0000 7f8521200640 1 --2- 172.21.15.28:0/2156833837 >> v2:172.21.15.191:3300/0 conn(0x7f851c150600 0x7f851c0c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.605+0000 7f8521c00640 1 -- 172.21.15.28:0/2156833837 >> v2:172.21.15.191:3300/0 conn(0x7f851c150600 msgr2=0x7f851c0c3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:29.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.605+0000 7f8521c00640 1 --2- 172.21.15.28:0/2156833837 >> v2:172.21.15.191:3300/0 conn(0x7f851c150600 0x7f851c0c3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.605+0000 7f8521c00640 1 -- 172.21.15.28:0/2156833837 >> v1:172.21.15.28:6789/0 conn(0x7f851c06ac20 legacy=0x7f851c159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.605+0000 7f8521c00640 1 -- 172.21.15.28:0/2156833837 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f851c0c5090 con 0x7f851c15a3c0 2024-09-13T16:48:29.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.606+0000 7f8521c00640 1 --2- 172.21.15.28:0/2156833837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f851c15a3c0 0x7f851c0c4040 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f851c04e860 tx=0x7f851402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4982ebf8be99b155 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:29.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.607+0000 7f851b400640 1 -- 172.21.15.28:0/2156833837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851402aea0 con 0x7f851c15a3c0 2024-09-13T16:48:29.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.607+0000 7f851b400640 1 -- 172.21.15.28:0/2156833837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f851402d050 con 0x7f851c15a3c0 2024-09-13T16:48:29.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.607+0000 7f851b400640 1 -- 172.21.15.28:0/2156833837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8514007960 con 0x7f851c15a3c0 2024-09-13T16:48:29.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.607+0000 7f8522600640 1 -- 172.21.15.28:0/2156833837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f851c15a3c0 msgr2=0x7f851c0c4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.607+0000 7f8522600640 1 --2- 172.21.15.28:0/2156833837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f851c15a3c0 0x7f851c0c4040 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f851c04e860 tx=0x7f851402a680 comp rx=0 tx=0).stop 2024-09-13T16:48:29.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.607+0000 7f8522600640 1 -- 172.21.15.28:0/2156833837 shutdown_connections 2024-09-13T16:48:29.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.607+0000 7f8522600640 1 --2- 172.21.15.28:0/2156833837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f851c15a3c0 0x7f851c0c4040 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.607+0000 7f8522600640 1 --2- 172.21.15.28:0/2156833837 >> v2:172.21.15.191:3300/0 conn(0x7f851c150600 0x7f851c0c3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.607+0000 7f8522600640 1 -- 172.21.15.28:0/2156833837 >> 172.21.15.28:0/2156833837 conn(0x7f851c069e80 msgr2=0x7f851c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:29.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.607+0000 7f8522600640 1 -- 172.21.15.28:0/2156833837 shutdown_connections 2024-09-13T16:48:29.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.607+0000 7f8522600640 1 -- 172.21.15.28:0/2156833837 wait complete. 2024-09-13T16:48:29.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.609+0000 7f8522600640 1 Processor -- start 2024-09-13T16:48:29.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.609+0000 7f8522600640 1 -- start start 2024-09-13T16:48:29.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.610+0000 7f8522600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f851c06ac20 0x7f851c1e1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.610+0000 7f8522600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f851c150600 0x7f851c1e5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.610+0000 7f8522600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f851c15a3c0 0x7f851c1587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.610+0000 7f8522600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851c159770 con 0x7f851c150600 2024-09-13T16:48:29.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.610+0000 7f8522600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851c14f700 con 0x7f851c06ac20 2024-09-13T16:48:29.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.610+0000 7f8522600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f851c14f460 con 0x7f851c15a3c0 2024-09-13T16:48:29.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.610+0000 7f851be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f851c150600 0x7f851c1e5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.610+0000 7f8521200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f851c06ac20 0x7f851c1e1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.610+0000 7f8521200640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f851c06ac20 0x7f851c1e1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:46722/0 (socket says 172.21.15.28:46722) 2024-09-13T16:48:29.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.610+0000 7f8521200640 1 -- 172.21.15.28:0/3756078337 learned_addr learned my addr 172.21.15.28:0/3756078337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:29.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.610+0000 7f8521c00640 1 --2- 172.21.15.28:0/3756078337 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f851c15a3c0 0x7f851c1587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.611+0000 7f851be00640 1 -- 172.21.15.28:0/3756078337 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f851c15a3c0 msgr2=0x7f851c1587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.611+0000 7f851be00640 1 --2- 172.21.15.28:0/3756078337 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f851c15a3c0 0x7f851c1587d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.611+0000 7f851be00640 1 -- 172.21.15.28:0/3756078337 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f851c06ac20 msgr2=0x7f851c1e1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.611+0000 7f851be00640 1 --2- 172.21.15.28:0/3756078337 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f851c06ac20 0x7f851c1e1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.611+0000 7f851be00640 1 -- 172.21.15.28:0/3756078337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8514009a70 con 0x7f851c150600 2024-09-13T16:48:29.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.611+0000 7f851be00640 1 --2- 172.21.15.28:0/3756078337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f851c150600 0x7f851c1e5120 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f850800c410 tx=0x7f8508001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:29.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.611+0000 7f8518c00640 1 -- 172.21.15.28:0/3756078337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8508005ce0 con 0x7f851c150600 2024-09-13T16:48:29.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.611+0000 7f8518c00640 1 -- 172.21.15.28:0/3756078337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8508005e40 con 0x7f851c150600 2024-09-13T16:48:29.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.611+0000 7f8518c00640 1 -- 172.21.15.28:0/3756078337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85080072d0 con 0x7f851c150600 2024-09-13T16:48:29.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.611+0000 7f8522600640 1 -- 172.21.15.28:0/3756078337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f851c1f16c0 con 0x7f851c150600 2024-09-13T16:48:29.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.611+0000 7f8522600640 1 -- 172.21.15.28:0/3756078337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f851c1f1c80 con 0x7f851c150600 2024-09-13T16:48:29.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.612+0000 7f8522600640 1 -- 172.21.15.28:0/3756078337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f851c0c6410 con 0x7f851c150600 2024-09-13T16:48:29.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.613+0000 7f8518c00640 1 -- 172.21.15.28:0/3756078337 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f85080074f0 con 0x7f851c150600 2024-09-13T16:48:29.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.614+0000 7f8518c00640 1 --2- 172.21.15.28:0/3756078337 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f84fc06ac00 0x7f84fc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.614+0000 7f8518c00640 1 -- 172.21.15.28:0/3756078337 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8508088850 con 0x7f851c150600 2024-09-13T16:48:29.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.614+0000 7f8518c00640 1 --2- 172.21.15.28:0/3756078337 >> [v2:172.21.15.191:6824/1477327720,v1:172.21.15.191:6825/1477327720] conn(0x7f84fc072f20 0x7f84fc0753a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.614+0000 7f8518c00640 1 -- 172.21.15.28:0/3756078337 --> [v2:172.21.15.191:6824/1477327720,v1:172.21.15.191:6825/1477327720] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f84fc075a70 con 0x7f84fc072f20 2024-09-13T16:48:29.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.614+0000 7f8518c00640 1 -- 172.21.15.28:0/3756078337 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f850808ae30 con 0x7f851c150600 2024-09-13T16:48:29.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.614+0000 7f8521200640 1 --2- 172.21.15.28:0/3756078337 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f84fc06ac00 0x7f84fc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.615+0000 7f8521c00640 1 --2- 172.21.15.28:0/3756078337 >> [v2:172.21.15.191:6824/1477327720,v1:172.21.15.191:6825/1477327720] conn(0x7f84fc072f20 0x7f84fc0753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.615+0000 7f8521200640 1 --2- 172.21.15.28:0/3756078337 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f84fc06ac00 0x7f84fc06d0c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8510009c00 tx=0x7f8510009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:29.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.615+0000 7f8521c00640 1 --2- 172.21.15.28:0/3756078337 >> [v2:172.21.15.191:6824/1477327720,v1:172.21.15.191:6825/1477327720] conn(0x7f84fc072f20 0x7f84fc0753a0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:29.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.616+0000 7f8518c00640 1 -- 172.21.15.28:0/3756078337 <== osd.7 v2:172.21.15.191:6824/1477327720 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f84fc075a70 con 0x7f84fc072f20 2024-09-13T16:48:29.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.629+0000 7f8522600640 1 -- 172.21.15.28:0/3756078337 --> [v2:172.21.15.191:6824/1477327720,v1:172.21.15.191:6825/1477327720] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f851c06e240 con 0x7f84fc072f20 2024-09-13T16:48:29.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.630+0000 7f8518c00640 1 -- 172.21.15.28:0/3756078337 <== osd.7 v2:172.21.15.191:6824/1477327720 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f8514009450 con 0x7f84fc072f20 2024-09-13T16:48:29.631 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T16:48:29.631 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T16:48:29.631 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T16:48:29.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.630+0000 7f8522600640 1 -- 172.21.15.28:0/3756078337 >> [v2:172.21.15.191:6824/1477327720,v1:172.21.15.191:6825/1477327720] conn(0x7f84fc072f20 msgr2=0x7f84fc0753a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.630+0000 7f8522600640 1 --2- 172.21.15.28:0/3756078337 >> [v2:172.21.15.191:6824/1477327720,v1:172.21.15.191:6825/1477327720] conn(0x7f84fc072f20 0x7f84fc0753a0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.630+0000 7f8522600640 1 -- 172.21.15.28:0/3756078337 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f84fc06ac00 msgr2=0x7f84fc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.630+0000 7f8522600640 1 --2- 172.21.15.28:0/3756078337 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f84fc06ac00 0x7f84fc06d0c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8510009c00 tx=0x7f8510009190 comp rx=0 tx=0).stop 2024-09-13T16:48:29.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.630+0000 7f8522600640 1 -- 172.21.15.28:0/3756078337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f851c150600 msgr2=0x7f851c1e5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.630+0000 7f8522600640 1 --2- 172.21.15.28:0/3756078337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f851c150600 0x7f851c1e5120 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f850800c410 tx=0x7f8508001e70 comp rx=0 tx=0).stop 2024-09-13T16:48:29.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.630+0000 7f8521c00640 1 -- 172.21.15.28:0/3756078337 reap_dead start 2024-09-13T16:48:29.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.630+0000 7f8522600640 1 -- 172.21.15.28:0/3756078337 shutdown_connections 2024-09-13T16:48:29.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.630+0000 7f8522600640 1 -- 172.21.15.28:0/3756078337 >> 172.21.15.28:0/3756078337 conn(0x7f851c069e80 msgr2=0x7f851c06cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:29.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.630+0000 7f8522600640 1 -- 172.21.15.28:0/3756078337 shutdown_connections 2024-09-13T16:48:29.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.630+0000 7f8522600640 1 -- 172.21.15.28:0/3756078337 wait complete. 2024-09-13T16:48:29.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-13T16:48:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.802+0000 7f0ec2800640 1 Processor -- start 2024-09-13T16:48:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.802+0000 7f0ec2800640 1 -- start start 2024-09-13T16:48:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.804+0000 7f0ec2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ebc0c73e0 0x7f0ebc0c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.804+0000 7f0ec2800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0ebc0c5a60 0x7f0ebc14f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.804+0000 7f0ec2800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ebc0c6f50 con 0x7f0ebc0c5140 2024-09-13T16:48:29.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.804+0000 7f0ec2800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ebc0c6940 con 0x7f0ebc0c73e0 2024-09-13T16:48:29.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.804+0000 7f0ec2800640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ebc0c1b50 con 0x7f0ebc0c5a60 2024-09-13T16:48:29.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.804+0000 7f0eb2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ebc0c73e0 0x7f0ebc0c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.804+0000 7f0ebb400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0ebc0c5140 0x7f0ebc159800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50338/0 (socket says 172.21.15.28:50338) 2024-09-13T16:48:29.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.804+0000 7f0eb2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ebc0c73e0 0x7f0ebc0c5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38222/0 (socket says 172.21.15.28:38222) 2024-09-13T16:48:29.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.804+0000 7f0ebb400640 1 -- 172.21.15.28:0/2841774543 learned_addr learned my addr 172.21.15.28:0/2841774543 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:48:29.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.804+0000 7f0ebbe00640 1 --2- 172.21.15.28:0/2841774543 >> v2:172.21.15.191:3300/0 conn(0x7f0ebc0c5a60 0x7f0ebc14f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.804+0000 7f0eb2a00640 1 -- 172.21.15.28:0/2841774543 >> v2:172.21.15.191:3300/0 conn(0x7f0ebc0c5a60 msgr2=0x7f0ebc14f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:29.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.804+0000 7f0eb2a00640 1 --2- 172.21.15.28:0/2841774543 >> v2:172.21.15.191:3300/0 conn(0x7f0ebc0c5a60 0x7f0ebc14f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.804+0000 7f0eb2a00640 1 -- 172.21.15.28:0/2841774543 >> v1:172.21.15.28:6789/0 conn(0x7f0ebc0c5140 legacy=0x7f0ebc159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.804+0000 7f0eb2a00640 1 -- 172.21.15.28:0/2841774543 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ebc0c1e30 con 0x7f0ebc0c73e0 2024-09-13T16:48:29.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.804+0000 7f0ebaa00640 1 -- 172.21.15.28:0/2841774543 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022286936 0 0) 0x7f0ebc0c6f50 con 0x7f0ebc0c5140 2024-09-13T16:48:29.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.805+0000 7f0eb2a00640 1 --2- 172.21.15.28:0/2841774543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0c73e0 0x7f0ebc0c5520 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0c6800 tx=0x7f0ea402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e7ae1f3117438 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:29.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.806+0000 7f0ebaa00640 1 -- 172.21.15.28:0/2841774543 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea402ae60 con 0x7f0ebc0c73e0 2024-09-13T16:48:29.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.806+0000 7f0ebaa00640 1 -- 172.21.15.28:0/2841774543 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea4032440 con 0x7f0ebc0c73e0 2024-09-13T16:48:29.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.806+0000 7f0ebaa00640 1 -- 172.21.15.28:0/2841774543 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea40327d0 con 0x7f0ebc0c73e0 2024-09-13T16:48:29.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.806+0000 7f0ec2800640 1 -- 172.21.15.28:0/2841774543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0c73e0 msgr2=0x7f0ebc0c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.806+0000 7f0ec2800640 1 --2- 172.21.15.28:0/2841774543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0c73e0 0x7f0ebc0c5520 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0c6800 tx=0x7f0ea402a110 comp rx=0 tx=0).stop 2024-09-13T16:48:29.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.806+0000 7f0ec2800640 1 -- 172.21.15.28:0/2841774543 shutdown_connections 2024-09-13T16:48:29.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.806+0000 7f0ec2800640 1 --2- 172.21.15.28:0/2841774543 >> v2:172.21.15.191:3300/0 conn(0x7f0ebc0c5a60 0x7f0ebc14f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.806+0000 7f0ec2800640 1 --2- 172.21.15.28:0/2841774543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0c73e0 0x7f0ebc0c5520 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.806+0000 7f0ec2800640 1 -- 172.21.15.28:0/2841774543 >> 172.21.15.28:0/2841774543 conn(0x7f0ebc0ca6b0 msgr2=0x7f0ebc0caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:29.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.806+0000 7f0ec2800640 1 -- 172.21.15.28:0/2841774543 shutdown_connections 2024-09-13T16:48:29.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.806+0000 7f0ec2800640 1 -- 172.21.15.28:0/2841774543 wait complete. 2024-09-13T16:48:29.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.808+0000 7f0ec2800640 1 Processor -- start 2024-09-13T16:48:29.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.808+0000 7f0ec2800640 1 -- start start 2024-09-13T16:48:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.809+0000 7f0ec2800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ebc0c5140 0x7f0ebc142720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.809+0000 7f0ec2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0c5a60 0x7f0ebc14b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.809+0000 7f0ec2800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ebc0c73e0 0x7f0ebc13dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.809+0000 7f0ec2800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc1f2730 con 0x7f0ebc0c5a60 2024-09-13T16:48:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.809+0000 7f0ec2800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc1f2200 con 0x7f0ebc0c5140 2024-09-13T16:48:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.809+0000 7f0ec2800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc14ac10 con 0x7f0ebc0c73e0 2024-09-13T16:48:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.809+0000 7f0ebb400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ebc0c5140 0x7f0ebc142720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.809+0000 7f0eb2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0c5a60 0x7f0ebc14b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.809+0000 7f0ebb400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ebc0c5140 0x7f0ebc142720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:46732/0 (socket says 172.21.15.28:46732) 2024-09-13T16:48:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.809+0000 7f0ebb400640 1 -- 172.21.15.28:0/1937765421 learned_addr learned my addr 172.21.15.28:0/1937765421 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.809+0000 7f0ebbe00640 1 --2- 172.21.15.28:0/1937765421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ebc0c73e0 0x7f0ebc13dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.810+0000 7f0ebb400640 1 -- 172.21.15.28:0/1937765421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ebc0c73e0 msgr2=0x7f0ebc13dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.810+0000 7f0ebb400640 1 --2- 172.21.15.28:0/1937765421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ebc0c73e0 0x7f0ebc13dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.810+0000 7f0ebb400640 1 -- 172.21.15.28:0/1937765421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0c5a60 msgr2=0x7f0ebc14b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.810+0000 7f0ebb400640 1 --2- 172.21.15.28:0/1937765421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0c5a60 0x7f0ebc14b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.810+0000 7f0ebb400640 1 -- 172.21.15.28:0/1937765421 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea4009400 con 0x7f0ebc0c5140 2024-09-13T16:48:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.810+0000 7f0ebbe00640 1 --2- 172.21.15.28:0/1937765421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ebc0c73e0 0x7f0ebc13dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.810+0000 7f0ebb400640 1 --2- 172.21.15.28:0/1937765421 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ebc0c5140 0x7f0ebc142720 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00c460 tx=0x7f0eac00c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.810+0000 7f0eb3e00640 1 -- 172.21.15.28:0/1937765421 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac0074e0 con 0x7f0ebc0c5140 2024-09-13T16:48:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.810+0000 7f0eb3e00640 1 -- 172.21.15.28:0/1937765421 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eac007660 con 0x7f0ebc0c5140 2024-09-13T16:48:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.810+0000 7f0eb3e00640 1 -- 172.21.15.28:0/1937765421 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac004a20 con 0x7f0ebc0c5140 2024-09-13T16:48:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.810+0000 7f0ec2800640 1 -- 172.21.15.28:0/1937765421 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ebc14a8e0 con 0x7f0ebc0c5140 2024-09-13T16:48:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.811+0000 7f0ec2800640 1 -- 172.21.15.28:0/1937765421 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ebc1f2450 con 0x7f0ebc0c5140 2024-09-13T16:48:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.812+0000 7f0ec2800640 1 -- 172.21.15.28:0/1937765421 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ebc14a740 con 0x7f0ebc0c5140 2024-09-13T16:48:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.813+0000 7f0eb3e00640 1 -- 172.21.15.28:0/1937765421 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0eac004b80 con 0x7f0ebc0c5140 2024-09-13T16:48:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.813+0000 7f0eb3e00640 1 --2- 172.21.15.28:0/1937765421 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0e9806ac30 0x7f0e9806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.814+0000 7f0eb3e00640 1 -- 172.21.15.28:0/1937765421 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0eac088570 con 0x7f0ebc0c5140 2024-09-13T16:48:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.814+0000 7f0eb2a00640 1 --2- 172.21.15.28:0/1937765421 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0e9806ac30 0x7f0e9806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.814+0000 7f0eb2a00640 1 --2- 172.21.15.28:0/1937765421 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0e9806ac30 0x7f0e9806d0f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4034000 tx=0x7f0ea402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.818+0000 7f0eb3e00640 1 -- 172.21.15.28:0/1937765421 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0ebc14a740 con 0x7f0ebc0c5140 2024-09-13T16:48:29.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.996+0000 7f0ec2800640 1 -- 172.21.15.28:0/1937765421 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ebc0caa90 con 0x7f0e9806ac30 2024-09-13T16:48:29.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.996+0000 7f0eb3e00640 1 -- 172.21.15.28:0/1937765421 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0ebc0caa90 con 0x7f0e9806ac30 2024-09-13T16:48:29.998 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 7 to deep-scrub 2024-09-13T16:48:30.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.998+0000 7f0ec2800640 1 -- 172.21.15.28:0/1937765421 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0e9806ac30 msgr2=0x7f0e9806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:30.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.998+0000 7f0ec2800640 1 --2- 172.21.15.28:0/1937765421 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0e9806ac30 0x7f0e9806d0f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4034000 tx=0x7f0ea402e000 comp rx=0 tx=0).stop 2024-09-13T16:48:30.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.998+0000 7f0ec2800640 1 -- 172.21.15.28:0/1937765421 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ebc0c5140 msgr2=0x7f0ebc142720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:30.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.998+0000 7f0ec2800640 1 --2- 172.21.15.28:0/1937765421 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ebc0c5140 0x7f0ebc142720 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00c460 tx=0x7f0eac00c920 comp rx=0 tx=0).stop 2024-09-13T16:48:30.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.998+0000 7f0ec2800640 1 -- 172.21.15.28:0/1937765421 shutdown_connections 2024-09-13T16:48:30.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.998+0000 7f0ec2800640 1 --2- 172.21.15.28:0/1937765421 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f0e9806ac30 0x7f0e9806d0f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:30.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.998+0000 7f0ec2800640 1 --2- 172.21.15.28:0/1937765421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0ebc0c73e0 0x7f0ebc13dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:30.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.998+0000 7f0ec2800640 1 --2- 172.21.15.28:0/1937765421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc0c5a60 0x7f0ebc14b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:30.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.998+0000 7f0ec2800640 1 --2- 172.21.15.28:0/1937765421 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0ebc0c5140 0x7f0ebc142720 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:30.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.998+0000 7f0ec2800640 1 -- 172.21.15.28:0/1937765421 >> 172.21.15.28:0/1937765421 conn(0x7f0ebc0ca6b0 msgr2=0x7f0ebc06dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:30.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.998+0000 7f0ec2800640 1 -- 172.21.15.28:0/1937765421 shutdown_connections 2024-09-13T16:48:30.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:29.998+0000 7f0ec2800640 1 -- 172.21.15.28:0/1937765421 wait complete. 2024-09-13T16:48:30.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:48:30.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.170+0000 7f64b4400640 1 Processor -- start 2024-09-13T16:48:30.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.170+0000 7f64b4400640 1 -- start start 2024-09-13T16:48:30.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.171+0000 7f64b4400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f64ac150ca0 0x7f64ac1568e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:30.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.171+0000 7f64b4400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64ac0c4d60 0x7f64ac0c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:30.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.171+0000 7f64b4400640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ac151080 con 0x7f64ac0c2250 2024-09-13T16:48:30.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.172+0000 7f64b4400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64ac0c2670 con 0x7f64ac150ca0 2024-09-13T16:48:30.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.172+0000 7f64b4400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac15ae40 con 0x7f64ac0c4d60 2024-09-13T16:48:30.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.172+0000 7f64b1a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f64ac150ca0 0x7f64ac1568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:30.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.172+0000 7f64b1a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f64ac150ca0 0x7f64ac1568e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38238/0 (socket says 172.21.15.28:38238) 2024-09-13T16:48:30.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.172+0000 7f64b1a00640 1 -- 172.21.15.28:0/2444022815 learned_addr learned my addr 172.21.15.28:0/2444022815 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:30.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.172+0000 7f64b2400640 1 --2- 172.21.15.28:0/2444022815 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64ac0c4d60 0x7f64ac0c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:30.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.172+0000 7f64b1a00640 1 -- 172.21.15.28:0/2444022815 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64ac0c4d60 msgr2=0x7f64ac0c51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:30.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.172+0000 7f64b1a00640 1 --2- 172.21.15.28:0/2444022815 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64ac0c4d60 0x7f64ac0c51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:30.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.172+0000 7f64b1a00640 1 -- 172.21.15.28:0/2444022815 >> v1:172.21.15.191:6789/0 conn(0x7f64ac0c2250 legacy=0x7f64ac15a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:30.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.172+0000 7f64b1a00640 1 -- 172.21.15.28:0/2444022815 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ac06a990 con 0x7f64ac150ca0 2024-09-13T16:48:30.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.172+0000 7f64abe00640 1 -- 172.21.15.28:0/2444022815 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358049914 0 0) 0x7f64ac151080 con 0x7f64ac0c2250 2024-09-13T16:48:30.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.173+0000 7f64b1a00640 1 --2- 172.21.15.28:0/2444022815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac150ca0 0x7f64ac1568e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f64a00098a0 tx=0x7f64a002a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5da9fa8f0e86994 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:30.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.174+0000 7f64abe00640 1 -- 172.21.15.28:0/2444022815 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0032440 con 0x7f64ac150ca0 2024-09-13T16:48:30.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.174+0000 7f64abe00640 1 -- 172.21.15.28:0/2444022815 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a00325a0 con 0x7f64ac150ca0 2024-09-13T16:48:30.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.174+0000 7f64abe00640 1 -- 172.21.15.28:0/2444022815 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0032930 con 0x7f64ac150ca0 2024-09-13T16:48:30.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.174+0000 7f64b4400640 1 -- 172.21.15.28:0/2444022815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac150ca0 msgr2=0x7f64ac1568e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:30.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.174+0000 7f64b4400640 1 --2- 172.21.15.28:0/2444022815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac150ca0 0x7f64ac1568e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f64a00098a0 tx=0x7f64a002a200 comp rx=0 tx=0).stop 2024-09-13T16:48:30.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.174+0000 7f64b4400640 1 -- 172.21.15.28:0/2444022815 shutdown_connections 2024-09-13T16:48:30.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.174+0000 7f64b4400640 1 --2- 172.21.15.28:0/2444022815 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64ac0c4d60 0x7f64ac0c51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:30.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.174+0000 7f64b4400640 1 --2- 172.21.15.28:0/2444022815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac150ca0 0x7f64ac1568e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:30.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.174+0000 7f64b4400640 1 -- 172.21.15.28:0/2444022815 >> 172.21.15.28:0/2444022815 conn(0x7f64ac066040 msgr2=0x7f64ac069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:30.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.174+0000 7f64b4400640 1 -- 172.21.15.28:0/2444022815 shutdown_connections 2024-09-13T16:48:30.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.174+0000 7f64b4400640 1 -- 172.21.15.28:0/2444022815 wait complete. 2024-09-13T16:48:30.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.176+0000 7f64b4400640 1 Processor -- start 2024-09-13T16:48:30.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.176+0000 7f64b4400640 1 -- start start 2024-09-13T16:48:30.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.177+0000 7f64b4400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac0c2250 0x7f64ac158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:30.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.177+0000 7f64b4400640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64ac0c4d60 0x7f64ac1dffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:30.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.177+0000 7f64b4400640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64ac150ca0 0x7f64ac0c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:30.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.177+0000 7f64b4400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac159bf0 con 0x7f64ac0c2250 2024-09-13T16:48:30.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.177+0000 7f64b4400640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac0c9710 con 0x7f64ac0c4d60 2024-09-13T16:48:30.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.177+0000 7f64b4400640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac0c9990 con 0x7f64ac150ca0 2024-09-13T16:48:30.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.177+0000 7f64b1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac0c2250 0x7f64ac158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:30.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.177+0000 7f64b1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac0c2250 0x7f64ac158730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38240/0 (socket says 172.21.15.28:38240) 2024-09-13T16:48:30.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.177+0000 7f64b1a00640 1 -- 172.21.15.28:0/99761693 learned_addr learned my addr 172.21.15.28:0/99761693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:30.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.177+0000 7f64b1000640 1 --2- 172.21.15.28:0/99761693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64ac0c4d60 0x7f64ac1dffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:30.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.177+0000 7f64b2400640 1 --2- 172.21.15.28:0/99761693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64ac150ca0 0x7f64ac0c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:30.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.178+0000 7f64b1a00640 1 -- 172.21.15.28:0/99761693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64ac150ca0 msgr2=0x7f64ac0c9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:30.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.178+0000 7f64b1a00640 1 --2- 172.21.15.28:0/99761693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64ac150ca0 0x7f64ac0c9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:30.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.178+0000 7f64b1a00640 1 -- 172.21.15.28:0/99761693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64ac0c4d60 msgr2=0x7f64ac1dffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:30.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.178+0000 7f64b1a00640 1 --2- 172.21.15.28:0/99761693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64ac0c4d60 0x7f64ac1dffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:30.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.178+0000 7f64b1a00640 1 -- 172.21.15.28:0/99761693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a00094d0 con 0x7f64ac0c2250 2024-09-13T16:48:30.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.178+0000 7f64b1a00640 1 --2- 172.21.15.28:0/99761693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac0c2250 0x7f64ac158730 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f64a00098a0 tx=0x7f64a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:30.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.178+0000 7f64a9600640 1 -- 172.21.15.28:0/99761693 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0032440 con 0x7f64ac0c2250 2024-09-13T16:48:30.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.178+0000 7f64a9600640 1 -- 172.21.15.28:0/99761693 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a0004250 con 0x7f64ac0c2250 2024-09-13T16:48:30.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.178+0000 7f64a9600640 1 -- 172.21.15.28:0/99761693 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0032e20 con 0x7f64ac0c2250 2024-09-13T16:48:30.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.178+0000 7f64b4400640 1 -- 172.21.15.28:0/99761693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ac1595a0 con 0x7f64ac0c2250 2024-09-13T16:48:30.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.179+0000 7f64b4400640 1 -- 172.21.15.28:0/99761693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ac0ca1b0 con 0x7f64ac0c2250 2024-09-13T16:48:30.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.180+0000 7f64b4400640 1 -- 172.21.15.28:0/99761693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ac0c1960 con 0x7f64ac0c2250 2024-09-13T16:48:30.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.180+0000 7f64a9600640 1 -- 172.21.15.28:0/99761693 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f64a0030980 con 0x7f64ac0c2250 2024-09-13T16:48:30.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.181+0000 7f64a9600640 1 --2- 172.21.15.28:0/99761693 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f648c06ac00 0x7f648c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:30.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.181+0000 7f64a9600640 1 -- 172.21.15.28:0/99761693 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f64a00aa4d0 con 0x7f64ac0c2250 2024-09-13T16:48:30.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.182+0000 7f64b1000640 1 --2- 172.21.15.28:0/99761693 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f648c06ac00 0x7f648c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:30.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.182+0000 7f64b1000640 1 --2- 172.21.15.28:0/99761693 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f648c06ac00 0x7f648c06d0c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f64ac0c1590 tx=0x7f6498009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:30.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.185+0000 7f64a9600640 1 -- 172.21.15.28:0/99761693 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f64a0075760 con 0x7f64ac0c2250 2024-09-13T16:48:30.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.360+0000 7f64b4400640 1 -- 172.21.15.28:0/99761693 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f64ac0c2e30 con 0x7f648c06ac00 2024-09-13T16:48:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.368+0000 7f64a9600640 1 -- 172.21.15.28:0/99761693 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287724 (secure 0 0 0) 0x7f649800f820 con 0x7f648c06ac00 2024-09-13T16:48:30.376 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:48:30.378 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T16:48:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.372+0000 7f64b4400640 1 -- 172.21.15.28:0/99761693 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f648c06ac00 msgr2=0x7f648c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.372+0000 7f64b4400640 1 --2- 172.21.15.28:0/99761693 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f648c06ac00 0x7f648c06d0c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f64ac0c1590 tx=0x7f6498009040 comp rx=0 tx=0).stop 2024-09-13T16:48:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.372+0000 7f64b4400640 1 -- 172.21.15.28:0/99761693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac0c2250 msgr2=0x7f64ac158730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.372+0000 7f64b4400640 1 --2- 172.21.15.28:0/99761693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac0c2250 0x7f64ac158730 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f64a00098a0 tx=0x7f64a00046c0 comp rx=0 tx=0).stop 2024-09-13T16:48:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.372+0000 7f64b4400640 1 -- 172.21.15.28:0/99761693 shutdown_connections 2024-09-13T16:48:30.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.372+0000 7f64b4400640 1 --2- 172.21.15.28:0/99761693 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f648c06ac00 0x7f648c06d0c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:30.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.372+0000 7f64b4400640 1 --2- 172.21.15.28:0/99761693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64ac150ca0 0x7f64ac0c9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:30.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.372+0000 7f64b4400640 1 --2- 172.21.15.28:0/99761693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64ac0c4d60 0x7f64ac1dffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:30.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.372+0000 7f64b4400640 1 --2- 172.21.15.28:0/99761693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac0c2250 0x7f64ac158730 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:30.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.372+0000 7f64b4400640 1 -- 172.21.15.28:0/99761693 >> 172.21.15.28:0/99761693 conn(0x7f64ac066040 msgr2=0x7f64ac14ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:30.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.372+0000 7f64b4400640 1 -- 172.21.15.28:0/99761693 shutdown_connections 2024-09-13T16:48:30.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:30.372+0000 7f64b4400640 1 -- 172.21.15.28:0/99761693 wait complete. 2024-09-13T16:48:30.390 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-09-13T16:48:29.654610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":962201,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":121,"num_read_kb":105,"num_write":9785,"num_write_kb":25168807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11758,"ondisk_log_size":11758,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":309116,"kb_used_data":2876,"kb_used_omap":12,"kb_used_meta":306163,"kb_avail":754665604,"statfs":{"total":773094113280,"available":772777578496,"internally_reserved":0,"allocated":2945024,"data_stored":2586795,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12743,"internal_metadata":313511481},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002920"},"pg_stats":[{"pgid":"3.3f","version":"22'187","reported_seq":201,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.334938+0000","last_change":"2024-09-13T16:43:54.987241+0000","last_active":"2024-09-13T16:48:22.334938+0000","last_peered":"2024-09-13T16:48:22.334938+0000","last_clean":"2024-09-13T16:48:22.334938+0000","last_became_active":"2024-09-13T16:43:54.986988+0000","last_became_peered":"2024-09-13T16:43:54.986988+0000","last_unstale":"2024-09-13T16:48:22.334938+0000","last_undegraded":"2024-09-13T16:48:22.334938+0000","last_fullsized":"2024-09-13T16:48:22.334938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"22'138","reported_seq":153,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.343644+0000","last_change":"2024-09-13T16:43:55.001141+0000","last_active":"2024-09-13T16:48:22.343644+0000","last_peered":"2024-09-13T16:48:22.343644+0000","last_clean":"2024-09-13T16:48:22.343644+0000","last_became_active":"2024-09-13T16:43:55.000757+0000","last_became_peered":"2024-09-13T16:43:55.000757+0000","last_unstale":"2024-09-13T16:48:22.343644+0000","last_undegraded":"2024-09-13T16:48:22.343644+0000","last_fullsized":"2024-09-13T16:48:22.343644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"22'212","reported_seq":227,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.339679+0000","last_change":"2024-09-13T16:43:54.986925+0000","last_active":"2024-09-13T16:48:22.339679+0000","last_peered":"2024-09-13T16:48:22.339679+0000","last_clean":"2024-09-13T16:48:22.339679+0000","last_became_active":"2024-09-13T16:43:54.986198+0000","last_became_peered":"2024-09-13T16:43:54.986198+0000","last_unstale":"2024-09-13T16:48:22.339679+0000","last_undegraded":"2024-09-13T16:48:22.339679+0000","last_fullsized":"2024-09-13T16:48:22.339679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"22'133","reported_seq":147,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.349293+0000","last_change":"2024-09-13T16:43:54.992249+0000","last_active":"2024-09-13T16:48:22.349293+0000","last_peered":"2024-09-13T16:48:22.349293+0000","last_clean":"2024-09-13T16:48:22.349293+0000","last_became_active":"2024-09-13T16:43:54.991804+0000","last_became_peered":"2024-09-13T16:43:54.991804+0000","last_unstale":"2024-09-13T16:48:22.349293+0000","last_undegraded":"2024-09-13T16:48:22.349293+0000","last_fullsized":"2024-09-13T16:48:22.349293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":270336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"22'194","reported_seq":207,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.336668+0000","last_change":"2024-09-13T16:43:54.987564+0000","last_active":"2024-09-13T16:48:22.336668+0000","last_peered":"2024-09-13T16:48:22.336668+0000","last_clean":"2024-09-13T16:48:22.336668+0000","last_became_active":"2024-09-13T16:43:54.987328+0000","last_became_peered":"2024-09-13T16:43:54.987328+0000","last_unstale":"2024-09-13T16:48:22.336668+0000","last_undegraded":"2024-09-13T16:48:22.336668+0000","last_fullsized":"2024-09-13T16:48:22.336668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:36.006730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"22'156","reported_seq":169,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.353470+0000","last_change":"2024-09-13T16:43:55.000019+0000","last_active":"2024-09-13T16:48:22.353470+0000","last_peered":"2024-09-13T16:48:22.353470+0000","last_clean":"2024-09-13T16:48:22.353470+0000","last_became_active":"2024-09-13T16:43:54.999752+0000","last_became_peered":"2024-09-13T16:43:54.999752+0000","last_unstale":"2024-09-13T16:48:22.353470+0000","last_undegraded":"2024-09-13T16:48:22.353470+0000","last_fullsized":"2024-09-13T16:48:22.353470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"22'175","reported_seq":188,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.337463+0000","last_change":"2024-09-13T16:43:54.999533+0000","last_active":"2024-09-13T16:48:22.337463+0000","last_peered":"2024-09-13T16:48:22.337463+0000","last_clean":"2024-09-13T16:48:22.337463+0000","last_became_active":"2024-09-13T16:43:54.999350+0000","last_became_peered":"2024-09-13T16:43:54.999350+0000","last_unstale":"2024-09-13T16:48:22.337463+0000","last_undegraded":"2024-09-13T16:48:22.337463+0000","last_fullsized":"2024-09-13T16:48:22.337463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:29.408901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"22'165","reported_seq":179,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.351084+0000","last_change":"2024-09-13T16:43:54.998526+0000","last_active":"2024-09-13T16:48:22.351084+0000","last_peered":"2024-09-13T16:48:22.351084+0000","last_clean":"2024-09-13T16:48:22.351084+0000","last_became_active":"2024-09-13T16:43:54.996309+0000","last_became_peered":"2024-09-13T16:43:54.996309+0000","last_unstale":"2024-09-13T16:48:22.351084+0000","last_undegraded":"2024-09-13T16:48:22.351084+0000","last_fullsized":"2024-09-13T16:48:22.351084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"22'155","reported_seq":168,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.349314+0000","last_change":"2024-09-13T16:43:54.993786+0000","last_active":"2024-09-13T16:48:22.349314+0000","last_peered":"2024-09-13T16:48:22.349314+0000","last_clean":"2024-09-13T16:48:22.349314+0000","last_became_active":"2024-09-13T16:43:54.993475+0000","last_became_peered":"2024-09-13T16:43:54.993475+0000","last_unstale":"2024-09-13T16:48:22.349314+0000","last_undegraded":"2024-09-13T16:48:22.349314+0000","last_fullsized":"2024-09-13T16:48:22.349314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"22'189","reported_seq":203,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.351743+0000","last_change":"2024-09-13T16:43:54.986272+0000","last_active":"2024-09-13T16:48:22.351743+0000","last_peered":"2024-09-13T16:48:22.351743+0000","last_clean":"2024-09-13T16:48:22.351743+0000","last_became_active":"2024-09-13T16:43:54.985979+0000","last_became_peered":"2024-09-13T16:43:54.985979+0000","last_unstale":"2024-09-13T16:48:22.351743+0000","last_undegraded":"2024-09-13T16:48:22.351743+0000","last_fullsized":"2024-09-13T16:48:22.351743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"22'174","reported_seq":188,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.351664+0000","last_change":"2024-09-13T16:43:54.986207+0000","last_active":"2024-09-13T16:48:22.351664+0000","last_peered":"2024-09-13T16:48:22.351664+0000","last_clean":"2024-09-13T16:48:22.351664+0000","last_became_active":"2024-09-13T16:43:54.985948+0000","last_became_peered":"2024-09-13T16:43:54.985948+0000","last_unstale":"2024-09-13T16:48:22.351664+0000","last_undegraded":"2024-09-13T16:48:22.351664+0000","last_fullsized":"2024-09-13T16:48:22.351664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"22'183","reported_seq":197,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.331315+0000","last_change":"2024-09-13T16:43:54.986636+0000","last_active":"2024-09-13T16:48:22.331315+0000","last_peered":"2024-09-13T16:48:22.331315+0000","last_clean":"2024-09-13T16:48:22.331315+0000","last_became_active":"2024-09-13T16:43:54.986161+0000","last_became_peered":"2024-09-13T16:43:54.986161+0000","last_unstale":"2024-09-13T16:48:22.331315+0000","last_undegraded":"2024-09-13T16:48:22.331315+0000","last_fullsized":"2024-09-13T16:48:22.331315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"22'170","reported_seq":184,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.338003+0000","last_change":"2024-09-13T16:43:54.997783+0000","last_active":"2024-09-13T16:48:22.338003+0000","last_peered":"2024-09-13T16:48:22.338003+0000","last_clean":"2024-09-13T16:48:22.338003+0000","last_became_active":"2024-09-13T16:43:54.997162+0000","last_became_peered":"2024-09-13T16:43:54.997162+0000","last_unstale":"2024-09-13T16:48:22.338003+0000","last_undegraded":"2024-09-13T16:48:22.338003+0000","last_fullsized":"2024-09-13T16:48:22.338003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"22'180","reported_seq":194,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.343360+0000","last_change":"2024-09-13T16:43:54.995262+0000","last_active":"2024-09-13T16:48:22.343360+0000","last_peered":"2024-09-13T16:48:22.343360+0000","last_clean":"2024-09-13T16:48:22.343360+0000","last_became_active":"2024-09-13T16:43:54.994891+0000","last_became_peered":"2024-09-13T16:43:54.994891+0000","last_unstale":"2024-09-13T16:48:22.343360+0000","last_undegraded":"2024-09-13T16:48:22.343360+0000","last_fullsized":"2024-09-13T16:48:22.343360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"22'192","reported_seq":206,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.332120+0000","last_change":"2024-09-13T16:43:54.981534+0000","last_active":"2024-09-13T16:48:22.332120+0000","last_peered":"2024-09-13T16:48:22.332120+0000","last_clean":"2024-09-13T16:48:22.332120+0000","last_became_active":"2024-09-13T16:43:54.981191+0000","last_became_peered":"2024-09-13T16:43:54.981191+0000","last_unstale":"2024-09-13T16:48:22.332120+0000","last_undegraded":"2024-09-13T16:48:22.332120+0000","last_fullsized":"2024-09-13T16:48:22.332120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"22'160","reported_seq":173,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.354097+0000","last_change":"2024-09-13T16:43:54.994157+0000","last_active":"2024-09-13T16:48:22.354097+0000","last_peered":"2024-09-13T16:48:22.354097+0000","last_clean":"2024-09-13T16:48:22.354097+0000","last_became_active":"2024-09-13T16:43:54.993915+0000","last_became_peered":"2024-09-13T16:43:54.993915+0000","last_unstale":"2024-09-13T16:48:22.354097+0000","last_undegraded":"2024-09-13T16:48:22.354097+0000","last_fullsized":"2024-09-13T16:48:22.354097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:37.028714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"22'175","reported_seq":188,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.349171+0000","last_change":"2024-09-13T16:43:54.987328+0000","last_active":"2024-09-13T16:48:22.349171+0000","last_peered":"2024-09-13T16:48:22.349171+0000","last_clean":"2024-09-13T16:48:22.349171+0000","last_became_active":"2024-09-13T16:43:54.987048+0000","last_became_peered":"2024-09-13T16:43:54.987048+0000","last_unstale":"2024-09-13T16:48:22.349171+0000","last_undegraded":"2024-09-13T16:48:22.349171+0000","last_fullsized":"2024-09-13T16:48:22.349171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:15.645550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"22'142","reported_seq":156,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.338941+0000","last_change":"2024-09-13T16:43:54.985628+0000","last_active":"2024-09-13T16:48:22.338941+0000","last_peered":"2024-09-13T16:48:22.338941+0000","last_clean":"2024-09-13T16:48:22.338941+0000","last_became_active":"2024-09-13T16:43:54.985378+0000","last_became_peered":"2024-09-13T16:43:54.985378+0000","last_unstale":"2024-09-13T16:48:22.338941+0000","last_undegraded":"2024-09-13T16:48:22.338941+0000","last_fullsized":"2024-09-13T16:48:22.338941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"22'157","reported_seq":171,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.339447+0000","last_change":"2024-09-13T16:43:54.996712+0000","last_active":"2024-09-13T16:48:22.339447+0000","last_peered":"2024-09-13T16:48:22.339447+0000","last_clean":"2024-09-13T16:48:22.339447+0000","last_became_active":"2024-09-13T16:43:54.996523+0000","last_became_peered":"2024-09-13T16:43:54.996523+0000","last_unstale":"2024-09-13T16:48:22.339447+0000","last_undegraded":"2024-09-13T16:48:22.339447+0000","last_fullsized":"2024-09-13T16:48:22.339447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:42.192657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"22'167","reported_seq":182,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:29.426083+0000","last_change":"2024-09-13T16:43:54.984879+0000","last_active":"2024-09-13T16:48:29.426083+0000","last_peered":"2024-09-13T16:48:29.426083+0000","last_clean":"2024-09-13T16:48:29.426083+0000","last_became_active":"2024-09-13T16:43:54.984595+0000","last_became_peered":"2024-09-13T16:43:54.984595+0000","last_unstale":"2024-09-13T16:48:29.426083+0000","last_undegraded":"2024-09-13T16:48:29.426083+0000","last_fullsized":"2024-09-13T16:48:29.426083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"22'167","reported_seq":182,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:25.991520+0000","last_change":"2024-09-13T16:43:54.996457+0000","last_active":"2024-09-13T16:48:25.991520+0000","last_peered":"2024-09-13T16:48:25.991520+0000","last_clean":"2024-09-13T16:48:25.991520+0000","last_became_active":"2024-09-13T16:43:54.996275+0000","last_became_peered":"2024-09-13T16:43:54.996275+0000","last_unstale":"2024-09-13T16:48:25.991520+0000","last_undegraded":"2024-09-13T16:48:25.991520+0000","last_fullsized":"2024-09-13T16:48:25.991520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"22'199","reported_seq":213,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.351241+0000","last_change":"2024-09-13T16:43:54.985368+0000","last_active":"2024-09-13T16:48:22.351241+0000","last_peered":"2024-09-13T16:48:22.351241+0000","last_clean":"2024-09-13T16:48:22.351241+0000","last_became_active":"2024-09-13T16:43:54.985130+0000","last_became_peered":"2024-09-13T16:43:54.985130+0000","last_unstale":"2024-09-13T16:48:22.351241+0000","last_undegraded":"2024-09-13T16:48:22.351241+0000","last_fullsized":"2024-09-13T16:48:22.351241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"22'158","reported_seq":173,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:27.707752+0000","last_change":"2024-09-13T16:43:54.991862+0000","last_active":"2024-09-13T16:48:27.707752+0000","last_peered":"2024-09-13T16:48:27.707752+0000","last_clean":"2024-09-13T16:48:27.707752+0000","last_became_active":"2024-09-13T16:43:54.991554+0000","last_became_peered":"2024-09-13T16:43:54.991554+0000","last_unstale":"2024-09-13T16:48:27.707752+0000","last_undegraded":"2024-09-13T16:48:27.707752+0000","last_fullsized":"2024-09-13T16:48:27.707752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"22'162","reported_seq":174,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.343130+0000","last_change":"2024-09-13T16:44:59.221091+0000","last_active":"2024-09-13T16:48:22.343130+0000","last_peered":"2024-09-13T16:48:22.343130+0000","last_clean":"2024-09-13T16:48:22.343130+0000","last_became_active":"2024-09-13T16:44:59.220762+0000","last_became_peered":"2024-09-13T16:44:59.220762+0000","last_unstale":"2024-09-13T16:48:22.343130+0000","last_undegraded":"2024-09-13T16:48:22.343130+0000","last_fullsized":"2024-09-13T16:48:22.343130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"22'170","reported_seq":185,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.340601+0000","last_change":"2024-09-13T16:43:54.991366+0000","last_active":"2024-09-13T16:48:22.340601+0000","last_peered":"2024-09-13T16:48:22.340601+0000","last_clean":"2024-09-13T16:48:22.340601+0000","last_became_active":"2024-09-13T16:43:54.991127+0000","last_became_peered":"2024-09-13T16:43:54.991127+0000","last_unstale":"2024-09-13T16:48:22.340601+0000","last_undegraded":"2024-09-13T16:48:22.340601+0000","last_fullsized":"2024-09-13T16:48:22.340601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"22'180","reported_seq":195,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.339009+0000","last_change":"2024-09-13T16:43:54.985615+0000","last_active":"2024-09-13T16:48:22.339009+0000","last_peered":"2024-09-13T16:48:22.339009+0000","last_clean":"2024-09-13T16:48:22.339009+0000","last_became_active":"2024-09-13T16:43:54.985299+0000","last_became_peered":"2024-09-13T16:43:54.985299+0000","last_unstale":"2024-09-13T16:48:22.339009+0000","last_undegraded":"2024-09-13T16:48:22.339009+0000","last_fullsized":"2024-09-13T16:48:22.339009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"22'147","reported_seq":162,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.342395+0000","last_change":"2024-09-13T16:43:54.993325+0000","last_active":"2024-09-13T16:48:22.342395+0000","last_peered":"2024-09-13T16:48:22.342395+0000","last_clean":"2024-09-13T16:48:22.342395+0000","last_became_active":"2024-09-13T16:43:54.993009+0000","last_became_peered":"2024-09-13T16:43:54.993009+0000","last_unstale":"2024-09-13T16:48:22.342395+0000","last_undegraded":"2024-09-13T16:48:22.342395+0000","last_fullsized":"2024-09-13T16:48:22.342395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"22'188","reported_seq":203,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.340933+0000","last_change":"2024-09-13T16:43:54.998229+0000","last_active":"2024-09-13T16:48:22.340933+0000","last_peered":"2024-09-13T16:48:22.340933+0000","last_clean":"2024-09-13T16:48:22.340933+0000","last_became_active":"2024-09-13T16:43:54.995774+0000","last_became_peered":"2024-09-13T16:43:54.995774+0000","last_unstale":"2024-09-13T16:48:22.340933+0000","last_undegraded":"2024-09-13T16:48:22.340933+0000","last_fullsized":"2024-09-13T16:48:22.340933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"22'172","reported_seq":187,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.353451+0000","last_change":"2024-09-13T16:43:55.002126+0000","last_active":"2024-09-13T16:48:22.353451+0000","last_peered":"2024-09-13T16:48:22.353451+0000","last_clean":"2024-09-13T16:48:22.353451+0000","last_became_active":"2024-09-13T16:43:55.001713+0000","last_became_peered":"2024-09-13T16:43:55.001713+0000","last_unstale":"2024-09-13T16:48:22.353451+0000","last_undegraded":"2024-09-13T16:48:22.353451+0000","last_fullsized":"2024-09-13T16:48:22.353451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"22'156","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.349712+0000","last_change":"2024-09-13T16:43:54.992745+0000","last_active":"2024-09-13T16:48:22.349712+0000","last_peered":"2024-09-13T16:48:22.349712+0000","last_clean":"2024-09-13T16:48:22.349712+0000","last_became_active":"2024-09-13T16:43:54.992508+0000","last_became_peered":"2024-09-13T16:43:54.992508+0000","last_unstale":"2024-09-13T16:48:22.349712+0000","last_undegraded":"2024-09-13T16:48:22.349712+0000","last_fullsized":"2024-09-13T16:48:22.349712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:23.913470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"22'143","reported_seq":158,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.351081+0000","last_change":"2024-09-13T16:43:54.990631+0000","last_active":"2024-09-13T16:48:22.351081+0000","last_peered":"2024-09-13T16:48:22.351081+0000","last_clean":"2024-09-13T16:48:22.351081+0000","last_became_active":"2024-09-13T16:43:54.986277+0000","last_became_peered":"2024-09-13T16:43:54.986277+0000","last_unstale":"2024-09-13T16:48:22.351081+0000","last_undegraded":"2024-09-13T16:48:22.351081+0000","last_fullsized":"2024-09-13T16:48:22.351081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"22'184","reported_seq":199,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.343604+0000","last_change":"2024-09-13T16:43:55.000059+0000","last_active":"2024-09-13T16:48:22.343604+0000","last_peered":"2024-09-13T16:48:22.343604+0000","last_clean":"2024-09-13T16:48:22.343604+0000","last_became_active":"2024-09-13T16:43:54.999778+0000","last_became_peered":"2024-09-13T16:43:54.999778+0000","last_unstale":"2024-09-13T16:48:22.343604+0000","last_undegraded":"2024-09-13T16:48:22.343604+0000","last_fullsized":"2024-09-13T16:48:22.343604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"22'219","reported_seq":233,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.337935+0000","last_change":"2024-09-13T16:43:54.988375+0000","last_active":"2024-09-13T16:48:22.337935+0000","last_peered":"2024-09-13T16:48:22.337935+0000","last_clean":"2024-09-13T16:48:22.337935+0000","last_became_active":"2024-09-13T16:43:54.988131+0000","last_became_peered":"2024-09-13T16:43:54.988131+0000","last_unstale":"2024-09-13T16:48:22.337935+0000","last_undegraded":"2024-09-13T16:48:22.337935+0000","last_fullsized":"2024-09-13T16:48:22.337935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'5","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:24.509156+0000","last_change":"2024-09-13T16:43:54.555042+0000","last_active":"2024-09-13T16:46:24.509156+0000","last_peered":"2024-09-13T16:46:24.509156+0000","last_clean":"2024-09-13T16:46:24.509156+0000","last_became_active":"2024-09-13T16:43:54.554697+0000","last_became_peered":"2024-09-13T16:43:54.554697+0000","last_unstale":"2024-09-13T16:46:24.509156+0000","last_undegraded":"2024-09-13T16:46:24.509156+0000","last_fullsized":"2024-09-13T16:46:24.509156+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"22'173","reported_seq":187,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.353773+0000","last_change":"2024-09-13T16:43:54.998755+0000","last_active":"2024-09-13T16:48:22.353773+0000","last_peered":"2024-09-13T16:48:22.353773+0000","last_clean":"2024-09-13T16:48:22.353773+0000","last_became_active":"2024-09-13T16:43:54.998399+0000","last_became_peered":"2024-09-13T16:43:54.998399+0000","last_unstale":"2024-09-13T16:48:22.353773+0000","last_undegraded":"2024-09-13T16:48:22.353773+0000","last_fullsized":"2024-09-13T16:48:22.353773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:24.508135+0000","last_change":"2024-09-13T16:43:54.553250+0000","last_active":"2024-09-13T16:46:24.508135+0000","last_peered":"2024-09-13T16:46:24.508135+0000","last_clean":"2024-09-13T16:46:24.508135+0000","last_became_active":"2024-09-13T16:43:54.552977+0000","last_became_peered":"2024-09-13T16:43:54.552977+0000","last_unstale":"2024-09-13T16:46:24.508135+0000","last_undegraded":"2024-09-13T16:46:24.508135+0000","last_fullsized":"2024-09-13T16:46:24.508135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"22'5","reported_seq":31,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:28.811005+0000","last_change":"2024-09-13T16:48:28.811089+0000","last_active":"2024-09-13T16:48:28.811005+0000","last_peered":"2024-09-13T16:48:28.811005+0000","last_clean":"2024-09-13T16:48:28.811005+0000","last_became_active":"2024-09-13T16:43:54.557722+0000","last_became_peered":"2024-09-13T16:43:54.557722+0000","last_unstale":"2024-09-13T16:48:28.811005+0000","last_undegraded":"2024-09-13T16:48:28.811005+0000","last_fullsized":"2024-09-13T16:48:28.811005+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'5","last_scrub_stamp":"2024-09-13T16:48:28.810988+0000","last_deep_scrub":"22'5","last_deep_scrub_stamp":"2024-09-13T16:48:28.810988+0000","last_clean_scrub_stamp":"2024-09-13T16:48:28.810988+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:12.536424+0000","scrub_duration":0.0035919440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"22'159","reported_seq":173,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.345331+0000","last_change":"2024-09-13T16:43:55.002041+0000","last_active":"2024-09-13T16:48:22.345331+0000","last_peered":"2024-09-13T16:48:22.345331+0000","last_clean":"2024-09-13T16:48:22.345331+0000","last_became_active":"2024-09-13T16:43:55.001512+0000","last_became_peered":"2024-09-13T16:43:55.001512+0000","last_unstale":"2024-09-13T16:48:22.345331+0000","last_undegraded":"2024-09-13T16:48:22.345331+0000","last_fullsized":"2024-09-13T16:48:22.345331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.383328+0000","last_change":"2024-09-13T16:43:54.550389+0000","last_active":"2024-09-13T16:44:59.383328+0000","last_peered":"2024-09-13T16:44:59.383328+0000","last_clean":"2024-09-13T16:44:59.383328+0000","last_became_active":"2024-09-13T16:43:54.550154+0000","last_became_peered":"2024-09-13T16:43:54.550154+0000","last_unstale":"2024-09-13T16:44:59.383328+0000","last_undegraded":"2024-09-13T16:44:59.383328+0000","last_fullsized":"2024-09-13T16:44:59.383328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"22'178","reported_seq":192,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.346756+0000","last_change":"2024-09-13T16:43:54.993377+0000","last_active":"2024-09-13T16:48:22.346756+0000","last_peered":"2024-09-13T16:48:22.346756+0000","last_clean":"2024-09-13T16:48:22.346756+0000","last_became_active":"2024-09-13T16:43:54.993036+0000","last_became_peered":"2024-09-13T16:43:54.993036+0000","last_unstale":"2024-09-13T16:48:22.346756+0000","last_undegraded":"2024-09-13T16:48:22.346756+0000","last_fullsized":"2024-09-13T16:48:22.346756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'5","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:19.507738+0000","last_change":"2024-09-13T16:43:54.555528+0000","last_active":"2024-09-13T16:46:19.507738+0000","last_peered":"2024-09-13T16:46:19.507738+0000","last_clean":"2024-09-13T16:46:19.507738+0000","last_became_active":"2024-09-13T16:43:54.555267+0000","last_became_peered":"2024-09-13T16:43:54.555267+0000","last_unstale":"2024-09-13T16:46:19.507738+0000","last_undegraded":"2024-09-13T16:46:19.507738+0000","last_fullsized":"2024-09-13T16:46:19.507738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"22'153","reported_seq":166,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.349653+0000","last_change":"2024-09-13T16:43:54.981706+0000","last_active":"2024-09-13T16:48:22.349653+0000","last_peered":"2024-09-13T16:48:22.349653+0000","last_clean":"2024-09-13T16:48:22.349653+0000","last_became_active":"2024-09-13T16:43:54.981429+0000","last_became_peered":"2024-09-13T16:43:54.981429+0000","last_unstale":"2024-09-13T16:48:22.349653+0000","last_undegraded":"2024-09-13T16:48:22.349653+0000","last_fullsized":"2024-09-13T16:48:22.349653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.382875+0000","last_change":"2024-09-13T16:43:54.554593+0000","last_active":"2024-09-13T16:44:59.382875+0000","last_peered":"2024-09-13T16:44:59.382875+0000","last_clean":"2024-09-13T16:44:59.382875+0000","last_became_active":"2024-09-13T16:43:54.554307+0000","last_became_peered":"2024-09-13T16:43:54.554307+0000","last_unstale":"2024-09-13T16:44:59.382875+0000","last_undegraded":"2024-09-13T16:44:59.382875+0000","last_fullsized":"2024-09-13T16:44:59.382875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:02.665918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"22'204","reported_seq":218,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.351329+0000","last_change":"2024-09-13T16:43:54.993485+0000","last_active":"2024-09-13T16:48:22.351329+0000","last_peered":"2024-09-13T16:48:22.351329+0000","last_clean":"2024-09-13T16:48:22.351329+0000","last_became_active":"2024-09-13T16:43:54.993226+0000","last_became_peered":"2024-09-13T16:43:54.993226+0000","last_unstale":"2024-09-13T16:48:22.351329+0000","last_undegraded":"2024-09-13T16:48:22.351329+0000","last_fullsized":"2024-09-13T16:48:22.351329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'11","reported_seq":26,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:10.127308+0000","last_change":"2024-09-13T16:43:54.562976+0000","last_active":"2024-09-13T16:48:10.127308+0000","last_peered":"2024-09-13T16:48:10.127308+0000","last_clean":"2024-09-13T16:48:10.127308+0000","last_became_active":"2024-09-13T16:43:54.560480+0000","last_became_peered":"2024-09-13T16:43:54.560480+0000","last_unstale":"2024-09-13T16:48:10.127308+0000","last_undegraded":"2024-09-13T16:48:10.127308+0000","last_fullsized":"2024-09-13T16:48:10.127308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"22'211","reported_seq":225,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.335710+0000","last_change":"2024-09-13T16:43:54.986459+0000","last_active":"2024-09-13T16:48:22.335710+0000","last_peered":"2024-09-13T16:48:22.335710+0000","last_clean":"2024-09-13T16:48:22.335710+0000","last_became_active":"2024-09-13T16:43:54.986083+0000","last_became_peered":"2024-09-13T16:43:54.986083+0000","last_unstale":"2024-09-13T16:48:22.335710+0000","last_undegraded":"2024-09-13T16:48:22.335710+0000","last_fullsized":"2024-09-13T16:48:22.335710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'167","reported_seq":182,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.355900+0000","last_change":"2024-09-13T16:43:54.565929+0000","last_active":"2024-09-13T16:48:22.355900+0000","last_peered":"2024-09-13T16:48:22.355900+0000","last_clean":"2024-09-13T16:48:22.355900+0000","last_became_active":"2024-09-13T16:43:54.565701+0000","last_became_peered":"2024-09-13T16:43:54.565701+0000","last_unstale":"2024-09-13T16:48:22.355900+0000","last_undegraded":"2024-09-13T16:48:22.355900+0000","last_fullsized":"2024-09-13T16:48:22.355900+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364698,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"22'241","reported_seq":254,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.344184+0000","last_change":"2024-09-13T16:43:54.989068+0000","last_active":"2024-09-13T16:48:22.344184+0000","last_peered":"2024-09-13T16:48:22.344184+0000","last_clean":"2024-09-13T16:48:22.344184+0000","last_became_active":"2024-09-13T16:43:54.988779+0000","last_became_peered":"2024-09-13T16:43:54.988779+0000","last_unstale":"2024-09-13T16:48:22.344184+0000","last_undegraded":"2024-09-13T16:48:22.344184+0000","last_fullsized":"2024-09-13T16:48:22.344184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:46.926157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":213,"num_write_kb":581632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'15","reported_seq":41,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:28.421797+0000","last_change":"2024-09-13T16:48:28.421871+0000","last_active":"2024-09-13T16:48:28.421797+0000","last_peered":"2024-09-13T16:48:28.421797+0000","last_clean":"2024-09-13T16:48:28.421797+0000","last_became_active":"2024-09-13T16:43:54.561077+0000","last_became_peered":"2024-09-13T16:43:54.561077+0000","last_unstale":"2024-09-13T16:48:28.421797+0000","last_undegraded":"2024-09-13T16:48:28.421797+0000","last_fullsized":"2024-09-13T16:48:28.421797+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-09-13T16:48:28.421778+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-09-13T16:48:28.421778+0000","last_clean_scrub_stamp":"2024-09-13T16:48:28.421778+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:09.651705+0000","scrub_duration":0.0051464910000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"22'162","reported_seq":176,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.339855+0000","last_change":"2024-09-13T16:43:54.999517+0000","last_active":"2024-09-13T16:48:22.339855+0000","last_peered":"2024-09-13T16:48:22.339855+0000","last_clean":"2024-09-13T16:48:22.339855+0000","last_became_active":"2024-09-13T16:43:54.999283+0000","last_became_peered":"2024-09-13T16:43:54.999283+0000","last_unstale":"2024-09-13T16:48:22.339855+0000","last_undegraded":"2024-09-13T16:48:22.339855+0000","last_fullsized":"2024-09-13T16:48:22.339855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.383950+0000","last_change":"2024-09-13T16:43:54.542871+0000","last_active":"2024-09-13T16:44:59.383950+0000","last_peered":"2024-09-13T16:44:59.383950+0000","last_clean":"2024-09-13T16:44:59.383950+0000","last_became_active":"2024-09-13T16:43:54.542548+0000","last_became_peered":"2024-09-13T16:43:54.542548+0000","last_unstale":"2024-09-13T16:44:59.383950+0000","last_undegraded":"2024-09-13T16:44:59.383950+0000","last_fullsized":"2024-09-13T16:44:59.383950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:29.697704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"22'195","reported_seq":208,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.338570+0000","last_change":"2024-09-13T16:43:54.989168+0000","last_active":"2024-09-13T16:48:22.338570+0000","last_peered":"2024-09-13T16:48:22.338570+0000","last_clean":"2024-09-13T16:48:22.338570+0000","last_became_active":"2024-09-13T16:43:54.988716+0000","last_became_peered":"2024-09-13T16:43:54.988716+0000","last_unstale":"2024-09-13T16:48:22.338570+0000","last_undegraded":"2024-09-13T16:48:22.338570+0000","last_fullsized":"2024-09-13T16:48:22.338570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:14.520247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'7","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.300077+0000","last_change":"2024-09-13T16:43:54.551551+0000","last_active":"2024-09-13T16:48:22.300077+0000","last_peered":"2024-09-13T16:48:22.300077+0000","last_clean":"2024-09-13T16:48:22.300077+0000","last_became_active":"2024-09-13T16:43:54.551293+0000","last_became_peered":"2024-09-13T16:43:54.551293+0000","last_unstale":"2024-09-13T16:48:22.300077+0000","last_undegraded":"2024-09-13T16:48:22.300077+0000","last_fullsized":"2024-09-13T16:48:22.300077+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":940,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"22'169","reported_seq":183,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.342309+0000","last_change":"2024-09-13T16:43:54.994516+0000","last_active":"2024-09-13T16:48:22.342309+0000","last_peered":"2024-09-13T16:48:22.342309+0000","last_clean":"2024-09-13T16:48:22.342309+0000","last_became_active":"2024-09-13T16:43:54.994245+0000","last_became_peered":"2024-09-13T16:43:54.994245+0000","last_unstale":"2024-09-13T16:48:22.342309+0000","last_undegraded":"2024-09-13T16:48:22.342309+0000","last_fullsized":"2024-09-13T16:48:22.342309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'18","reported_seq":33,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:10.125365+0000","last_change":"2024-09-13T16:43:54.562232+0000","last_active":"2024-09-13T16:48:10.125365+0000","last_peered":"2024-09-13T16:48:10.125365+0000","last_clean":"2024-09-13T16:48:10.125365+0000","last_became_active":"2024-09-13T16:43:54.561939+0000","last_became_peered":"2024-09-13T16:43:54.561939+0000","last_unstale":"2024-09-13T16:48:10.125365+0000","last_undegraded":"2024-09-13T16:48:10.125365+0000","last_fullsized":"2024-09-13T16:48:10.125365+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"22'175","reported_seq":188,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.337087+0000","last_change":"2024-09-13T16:43:54.988439+0000","last_active":"2024-09-13T16:48:22.337087+0000","last_peered":"2024-09-13T16:48:22.337087+0000","last_clean":"2024-09-13T16:48:22.337087+0000","last_became_active":"2024-09-13T16:43:54.988172+0000","last_became_peered":"2024-09-13T16:43:54.988172+0000","last_unstale":"2024-09-13T16:48:22.337087+0000","last_undegraded":"2024-09-13T16:48:22.337087+0000","last_fullsized":"2024-09-13T16:48:22.337087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:55.871370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'5","reported_seq":32,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:28.623118+0000","last_change":"2024-09-13T16:48:28.623229+0000","last_active":"2024-09-13T16:48:28.623118+0000","last_peered":"2024-09-13T16:48:28.623118+0000","last_clean":"2024-09-13T16:48:28.623118+0000","last_became_active":"2024-09-13T16:43:54.559111+0000","last_became_peered":"2024-09-13T16:43:54.559111+0000","last_unstale":"2024-09-13T16:48:28.623118+0000","last_undegraded":"2024-09-13T16:48:28.623118+0000","last_fullsized":"2024-09-13T16:48:28.623118+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'5","last_scrub_stamp":"2024-09-13T16:48:28.623093+0000","last_deep_scrub":"22'5","last_deep_scrub_stamp":"2024-09-13T16:48:28.623093+0000","last_clean_scrub_stamp":"2024-09-13T16:48:28.623093+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:43.512775+0000","scrub_duration":0.0030479550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'156","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.337418+0000","last_change":"2024-09-13T16:43:54.988313+0000","last_active":"2024-09-13T16:48:22.337418+0000","last_peered":"2024-09-13T16:48:22.337418+0000","last_clean":"2024-09-13T16:48:22.337418+0000","last_became_active":"2024-09-13T16:43:54.988059+0000","last_became_peered":"2024-09-13T16:43:54.988059+0000","last_unstale":"2024-09-13T16:48:22.337418+0000","last_undegraded":"2024-09-13T16:48:22.337418+0000","last_fullsized":"2024-09-13T16:48:22.337418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.618840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:24.526821+0000","last_change":"2024-09-13T16:43:54.566410+0000","last_active":"2024-09-13T16:48:24.526821+0000","last_peered":"2024-09-13T16:48:24.526821+0000","last_clean":"2024-09-13T16:48:24.526821+0000","last_became_active":"2024-09-13T16:43:54.565861+0000","last_became_peered":"2024-09-13T16:43:54.565861+0000","last_unstale":"2024-09-13T16:48:24.526821+0000","last_undegraded":"2024-09-13T16:48:24.526821+0000","last_fullsized":"2024-09-13T16:48:24.526821+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"22'136","reported_seq":148,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.339101+0000","last_change":"2024-09-13T16:44:58.956863+0000","last_active":"2024-09-13T16:48:22.339101+0000","last_peered":"2024-09-13T16:48:22.339101+0000","last_clean":"2024-09-13T16:48:22.339101+0000","last_became_active":"2024-09-13T16:44:58.956557+0000","last_became_peered":"2024-09-13T16:44:58.956557+0000","last_unstale":"2024-09-13T16:48:22.339101+0000","last_undegraded":"2024-09-13T16:48:22.339101+0000","last_fullsized":"2024-09-13T16:48:22.339101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":278528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:24.529744+0000","last_change":"2024-09-13T16:43:54.559723+0000","last_active":"2024-09-13T16:48:24.529744+0000","last_peered":"2024-09-13T16:48:24.529744+0000","last_clean":"2024-09-13T16:48:24.529744+0000","last_became_active":"2024-09-13T16:43:54.559353+0000","last_became_peered":"2024-09-13T16:43:54.559353+0000","last_unstale":"2024-09-13T16:48:24.529744+0000","last_undegraded":"2024-09-13T16:48:24.529744+0000","last_fullsized":"2024-09-13T16:48:24.529744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"22'209","reported_seq":223,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.349215+0000","last_change":"2024-09-13T16:43:54.992511+0000","last_active":"2024-09-13T16:48:22.349215+0000","last_peered":"2024-09-13T16:48:22.349215+0000","last_clean":"2024-09-13T16:48:22.349215+0000","last_became_active":"2024-09-13T16:43:54.992249+0000","last_became_peered":"2024-09-13T16:43:54.992249+0000","last_unstale":"2024-09-13T16:48:22.349215+0000","last_undegraded":"2024-09-13T16:48:22.349215+0000","last_fullsized":"2024-09-13T16:48:22.349215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:27:38.850262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"20'5","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:24.526796+0000","last_change":"2024-09-13T16:43:54.551390+0000","last_active":"2024-09-13T16:48:24.526796+0000","last_peered":"2024-09-13T16:48:24.526796+0000","last_clean":"2024-09-13T16:48:24.526796+0000","last_became_active":"2024-09-13T16:43:54.547074+0000","last_became_peered":"2024-09-13T16:43:54.547074+0000","last_unstale":"2024-09-13T16:48:24.526796+0000","last_undegraded":"2024-09-13T16:48:24.526796+0000","last_fullsized":"2024-09-13T16:48:24.526796+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"22'172","reported_seq":186,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.350021+0000","last_change":"2024-09-13T16:43:54.990458+0000","last_active":"2024-09-13T16:48:22.350021+0000","last_peered":"2024-09-13T16:48:22.350021+0000","last_clean":"2024-09-13T16:48:22.350021+0000","last_became_active":"2024-09-13T16:43:54.990196+0000","last_became_peered":"2024-09-13T16:43:54.990196+0000","last_unstale":"2024-09-13T16:48:22.350021+0000","last_undegraded":"2024-09-13T16:48:22.350021+0000","last_fullsized":"2024-09-13T16:48:22.350021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:54.505478+0000","last_change":"2024-09-13T16:43:51.914639+0000","last_active":"2024-09-13T16:45:54.505478+0000","last_peered":"2024-09-13T16:45:54.505478+0000","last_clean":"2024-09-13T16:45:54.505478+0000","last_became_active":"2024-09-13T16:43:51.913797+0000","last_became_peered":"2024-09-13T16:43:51.913797+0000","last_unstale":"2024-09-13T16:45:54.505478+0000","last_undegraded":"2024-09-13T16:45:54.505478+0000","last_fullsized":"2024-09-13T16:45:54.505478+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:50.498116+0000","last_clean_scrub_stamp":"2024-09-13T16:43:50.498116+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:12.730911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:00.501470+0000","last_change":"2024-09-13T16:43:54.564153+0000","last_active":"2024-09-13T16:46:00.501470+0000","last_peered":"2024-09-13T16:46:00.501470+0000","last_clean":"2024-09-13T16:46:00.501470+0000","last_became_active":"2024-09-13T16:43:54.563949+0000","last_became_peered":"2024-09-13T16:43:54.563949+0000","last_unstale":"2024-09-13T16:46:00.501470+0000","last_undegraded":"2024-09-13T16:46:00.501470+0000","last_fullsized":"2024-09-13T16:46:00.501470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"22'215","reported_seq":227,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.351788+0000","last_change":"2024-09-13T16:44:59.387015+0000","last_active":"2024-09-13T16:48:22.351788+0000","last_peered":"2024-09-13T16:48:22.351788+0000","last_clean":"2024-09-13T16:48:22.351788+0000","last_became_active":"2024-09-13T16:44:59.386756+0000","last_became_peered":"2024-09-13T16:44:59.386756+0000","last_unstale":"2024-09-13T16:48:22.351788+0000","last_undegraded":"2024-09-13T16:48:22.351788+0000","last_fullsized":"2024-09-13T16:48:22.351788+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":186,"num_write_kb":507904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:54.505444+0000","last_change":"2024-09-13T16:43:54.564804+0000","last_active":"2024-09-13T16:45:54.505444+0000","last_peered":"2024-09-13T16:45:54.505444+0000","last_clean":"2024-09-13T16:45:54.505444+0000","last_became_active":"2024-09-13T16:43:54.564605+0000","last_became_peered":"2024-09-13T16:43:54.564605+0000","last_unstale":"2024-09-13T16:45:54.505444+0000","last_undegraded":"2024-09-13T16:45:54.505444+0000","last_fullsized":"2024-09-13T16:45:54.505444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:12.681134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"22'192","reported_seq":207,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.337565+0000","last_change":"2024-09-13T16:43:54.997884+0000","last_active":"2024-09-13T16:48:22.337565+0000","last_peered":"2024-09-13T16:48:22.337565+0000","last_clean":"2024-09-13T16:48:22.337565+0000","last_became_active":"2024-09-13T16:43:54.997510+0000","last_became_peered":"2024-09-13T16:43:54.997510+0000","last_unstale":"2024-09-13T16:48:22.337565+0000","last_undegraded":"2024-09-13T16:48:22.337565+0000","last_fullsized":"2024-09-13T16:48:22.337565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'13","reported_seq":34,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.300200+0000","last_change":"2024-09-13T16:43:54.566294+0000","last_active":"2024-09-13T16:48:22.300200+0000","last_peered":"2024-09-13T16:48:22.300200+0000","last_clean":"2024-09-13T16:48:22.300200+0000","last_became_active":"2024-09-13T16:43:54.566023+0000","last_became_peered":"2024-09-13T16:43:54.566023+0000","last_unstale":"2024-09-13T16:48:22.300200+0000","last_undegraded":"2024-09-13T16:48:22.300200+0000","last_fullsized":"2024-09-13T16:48:22.300200+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"22'167","reported_seq":182,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.336199+0000","last_change":"2024-09-13T16:43:54.981116+0000","last_active":"2024-09-13T16:48:22.336199+0000","last_peered":"2024-09-13T16:48:22.336199+0000","last_clean":"2024-09-13T16:48:22.336199+0000","last_became_active":"2024-09-13T16:43:54.980766+0000","last_became_peered":"2024-09-13T16:43:54.980766+0000","last_unstale":"2024-09-13T16:48:22.336199+0000","last_undegraded":"2024-09-13T16:48:22.336199+0000","last_fullsized":"2024-09-13T16:48:22.336199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:00.501452+0000","last_change":"2024-09-13T16:43:54.563716+0000","last_active":"2024-09-13T16:46:00.501452+0000","last_peered":"2024-09-13T16:46:00.501452+0000","last_clean":"2024-09-13T16:46:00.501452+0000","last_became_active":"2024-09-13T16:43:54.563522+0000","last_became_peered":"2024-09-13T16:43:54.563522+0000","last_unstale":"2024-09-13T16:46:00.501452+0000","last_undegraded":"2024-09-13T16:46:00.501452+0000","last_fullsized":"2024-09-13T16:46:00.501452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"22'163","reported_seq":178,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.345962+0000","last_change":"2024-09-13T16:43:54.999051+0000","last_active":"2024-09-13T16:48:22.345962+0000","last_peered":"2024-09-13T16:48:22.345962+0000","last_clean":"2024-09-13T16:48:22.345962+0000","last_became_active":"2024-09-13T16:43:54.998803+0000","last_became_peered":"2024-09-13T16:43:54.998803+0000","last_unstale":"2024-09-13T16:48:22.345962+0000","last_undegraded":"2024-09-13T16:48:22.345962+0000","last_fullsized":"2024-09-13T16:48:22.345962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:04.527484+0000","last_change":"2024-09-13T16:43:54.569945+0000","last_active":"2024-09-13T16:48:04.527484+0000","last_peered":"2024-09-13T16:48:04.527484+0000","last_clean":"2024-09-13T16:48:04.527484+0000","last_became_active":"2024-09-13T16:43:54.568846+0000","last_became_peered":"2024-09-13T16:43:54.568846+0000","last_unstale":"2024-09-13T16:48:04.527484+0000","last_undegraded":"2024-09-13T16:48:04.527484+0000","last_fullsized":"2024-09-13T16:48:04.527484+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"22'179","reported_seq":193,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.349059+0000","last_change":"2024-09-13T16:43:54.992082+0000","last_active":"2024-09-13T16:48:22.349059+0000","last_peered":"2024-09-13T16:48:22.349059+0000","last_clean":"2024-09-13T16:48:22.349059+0000","last_became_active":"2024-09-13T16:43:54.991659+0000","last_became_peered":"2024-09-13T16:43:54.991659+0000","last_unstale":"2024-09-13T16:48:22.349059+0000","last_undegraded":"2024-09-13T16:48:22.349059+0000","last_fullsized":"2024-09-13T16:48:22.349059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:14.795918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:45:59.506045+0000","last_change":"2024-09-13T16:43:54.569111+0000","last_active":"2024-09-13T16:45:59.506045+0000","last_peered":"2024-09-13T16:45:59.506045+0000","last_clean":"2024-09-13T16:45:59.506045+0000","last_became_active":"2024-09-13T16:43:54.568903+0000","last_became_peered":"2024-09-13T16:43:54.568903+0000","last_unstale":"2024-09-13T16:45:59.506045+0000","last_undegraded":"2024-09-13T16:45:59.506045+0000","last_fullsized":"2024-09-13T16:45:59.506045+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:35.629486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"22'206","reported_seq":221,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.341162+0000","last_change":"2024-09-13T16:43:54.980100+0000","last_active":"2024-09-13T16:48:22.341162+0000","last_peered":"2024-09-13T16:48:22.341162+0000","last_clean":"2024-09-13T16:48:22.341162+0000","last_became_active":"2024-09-13T16:43:54.979584+0000","last_became_peered":"2024-09-13T16:43:54.979584+0000","last_unstale":"2024-09-13T16:48:22.341162+0000","last_undegraded":"2024-09-13T16:48:22.341162+0000","last_fullsized":"2024-09-13T16:48:22.341162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'21","reported_seq":37,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:24.014636+0000","last_change":"2024-09-13T16:43:54.560710+0000","last_active":"2024-09-13T16:48:24.014636+0000","last_peered":"2024-09-13T16:48:24.014636+0000","last_clean":"2024-09-13T16:48:24.014636+0000","last_became_active":"2024-09-13T16:43:54.560030+0000","last_became_peered":"2024-09-13T16:43:54.560030+0000","last_unstale":"2024-09-13T16:48:24.014636+0000","last_undegraded":"2024-09-13T16:48:24.014636+0000","last_fullsized":"2024-09-13T16:48:24.014636+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'196","reported_seq":211,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.340984+0000","last_change":"2024-09-13T16:43:54.990812+0000","last_active":"2024-09-13T16:48:22.340984+0000","last_peered":"2024-09-13T16:48:22.340984+0000","last_clean":"2024-09-13T16:48:22.340984+0000","last_became_active":"2024-09-13T16:43:54.990440+0000","last_became_peered":"2024-09-13T16:43:54.990440+0000","last_unstale":"2024-09-13T16:48:22.340984+0000","last_undegraded":"2024-09-13T16:48:22.340984+0000","last_fullsized":"2024-09-13T16:48:22.340984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'8","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:14.520204+0000","last_change":"2024-09-13T16:43:54.558439+0000","last_active":"2024-09-13T16:48:14.520204+0000","last_peered":"2024-09-13T16:48:14.520204+0000","last_clean":"2024-09-13T16:48:14.520204+0000","last_became_active":"2024-09-13T16:43:54.558194+0000","last_became_peered":"2024-09-13T16:43:54.558194+0000","last_unstale":"2024-09-13T16:48:14.520204+0000","last_undegraded":"2024-09-13T16:48:14.520204+0000","last_fullsized":"2024-09-13T16:48:14.520204+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"22'161","reported_seq":176,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.348192+0000","last_change":"2024-09-13T16:43:54.995534+0000","last_active":"2024-09-13T16:48:22.348192+0000","last_peered":"2024-09-13T16:48:22.348192+0000","last_clean":"2024-09-13T16:48:22.348192+0000","last_became_active":"2024-09-13T16:43:54.995397+0000","last_became_peered":"2024-09-13T16:43:54.995397+0000","last_unstale":"2024-09-13T16:48:22.348192+0000","last_undegraded":"2024-09-13T16:48:22.348192+0000","last_fullsized":"2024-09-13T16:48:22.348192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:04.506432+0000","last_change":"2024-09-13T16:43:54.569085+0000","last_active":"2024-09-13T16:46:04.506432+0000","last_peered":"2024-09-13T16:46:04.506432+0000","last_clean":"2024-09-13T16:46:04.506432+0000","last_became_active":"2024-09-13T16:43:54.568892+0000","last_became_peered":"2024-09-13T16:43:54.568892+0000","last_unstale":"2024-09-13T16:46:04.506432+0000","last_undegraded":"2024-09-13T16:46:04.506432+0000","last_fullsized":"2024-09-13T16:46:04.506432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:08.334061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"22'198","reported_seq":213,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.336562+0000","last_change":"2024-09-13T16:43:54.984120+0000","last_active":"2024-09-13T16:48:22.336562+0000","last_peered":"2024-09-13T16:48:22.336562+0000","last_clean":"2024-09-13T16:48:22.336562+0000","last_became_active":"2024-09-13T16:43:54.983886+0000","last_became_peered":"2024-09-13T16:43:54.983886+0000","last_unstale":"2024-09-13T16:48:22.336562+0000","last_undegraded":"2024-09-13T16:48:22.336562+0000","last_fullsized":"2024-09-13T16:48:22.336562+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:14.528614+0000","last_change":"2024-09-13T16:43:54.555092+0000","last_active":"2024-09-13T16:48:14.528614+0000","last_peered":"2024-09-13T16:48:14.528614+0000","last_clean":"2024-09-13T16:48:14.528614+0000","last_became_active":"2024-09-13T16:43:54.554230+0000","last_became_peered":"2024-09-13T16:43:54.554230+0000","last_unstale":"2024-09-13T16:48:14.528614+0000","last_undegraded":"2024-09-13T16:48:14.528614+0000","last_fullsized":"2024-09-13T16:48:14.528614+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"22'178","reported_seq":193,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.345164+0000","last_change":"2024-09-13T16:43:54.984588+0000","last_active":"2024-09-13T16:48:22.345164+0000","last_peered":"2024-09-13T16:48:22.345164+0000","last_clean":"2024-09-13T16:48:22.345164+0000","last_became_active":"2024-09-13T16:43:54.984221+0000","last_became_peered":"2024-09-13T16:43:54.984221+0000","last_unstale":"2024-09-13T16:48:22.345164+0000","last_undegraded":"2024-09-13T16:48:22.345164+0000","last_fullsized":"2024-09-13T16:48:22.345164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:14.528631+0000","last_change":"2024-09-13T16:43:54.562646+0000","last_active":"2024-09-13T16:48:14.528631+0000","last_peered":"2024-09-13T16:48:14.528631+0000","last_clean":"2024-09-13T16:48:14.528631+0000","last_became_active":"2024-09-13T16:43:54.556888+0000","last_became_peered":"2024-09-13T16:43:54.556888+0000","last_unstale":"2024-09-13T16:48:14.528631+0000","last_undegraded":"2024-09-13T16:48:14.528631+0000","last_fullsized":"2024-09-13T16:48:14.528631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"22'217","reported_seq":232,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:27.707753+0000","last_change":"2024-09-13T16:43:54.999328+0000","last_active":"2024-09-13T16:48:27.707753+0000","last_peered":"2024-09-13T16:48:27.707753+0000","last_clean":"2024-09-13T16:48:27.707753+0000","last_became_active":"2024-09-13T16:43:54.999101+0000","last_became_peered":"2024-09-13T16:43:54.999101+0000","last_unstale":"2024-09-13T16:48:27.707753+0000","last_undegraded":"2024-09-13T16:48:27.707753+0000","last_fullsized":"2024-09-13T16:48:27.707753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'5","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:13.068914+0000","last_change":"2024-09-13T16:43:54.559214+0000","last_active":"2024-09-13T16:46:13.068914+0000","last_peered":"2024-09-13T16:46:13.068914+0000","last_clean":"2024-09-13T16:46:13.068914+0000","last_became_active":"2024-09-13T16:43:54.558233+0000","last_became_peered":"2024-09-13T16:43:54.558233+0000","last_unstale":"2024-09-13T16:46:13.068914+0000","last_undegraded":"2024-09-13T16:46:13.068914+0000","last_fullsized":"2024-09-13T16:46:13.068914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:50.954190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"22'190","reported_seq":205,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.327923+0000","last_change":"2024-09-13T16:43:54.986565+0000","last_active":"2024-09-13T16:48:22.327923+0000","last_peered":"2024-09-13T16:48:22.327923+0000","last_clean":"2024-09-13T16:48:22.327923+0000","last_became_active":"2024-09-13T16:43:54.986084+0000","last_became_peered":"2024-09-13T16:43:54.986084+0000","last_unstale":"2024-09-13T16:48:22.327923+0000","last_undegraded":"2024-09-13T16:48:22.327923+0000","last_fullsized":"2024-09-13T16:48:22.327923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"22'165","reported_seq":178,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.349946+0000","last_change":"2024-09-13T16:43:54.982611+0000","last_active":"2024-09-13T16:48:22.349946+0000","last_peered":"2024-09-13T16:48:22.349946+0000","last_clean":"2024-09-13T16:48:22.349946+0000","last_became_active":"2024-09-13T16:43:54.982338+0000","last_became_peered":"2024-09-13T16:43:54.982338+0000","last_unstale":"2024-09-13T16:48:22.349946+0000","last_undegraded":"2024-09-13T16:48:22.349946+0000","last_fullsized":"2024-09-13T16:48:22.349946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:17.664661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:44:59.382688+0000","last_change":"2024-09-13T16:43:54.558799+0000","last_active":"2024-09-13T16:44:59.382688+0000","last_peered":"2024-09-13T16:44:59.382688+0000","last_clean":"2024-09-13T16:44:59.382688+0000","last_became_active":"2024-09-13T16:43:54.558332+0000","last_became_peered":"2024-09-13T16:43:54.558332+0000","last_unstale":"2024-09-13T16:44:59.382688+0000","last_undegraded":"2024-09-13T16:44:59.382688+0000","last_fullsized":"2024-09-13T16:44:59.382688+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:19.201506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'152","reported_seq":164,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.342883+0000","last_change":"2024-09-13T16:44:58.957875+0000","last_active":"2024-09-13T16:48:22.342883+0000","last_peered":"2024-09-13T16:48:22.342883+0000","last_clean":"2024-09-13T16:48:22.342883+0000","last_became_active":"2024-09-13T16:44:58.957568+0000","last_became_peered":"2024-09-13T16:44:58.957568+0000","last_unstale":"2024-09-13T16:48:22.342883+0000","last_undegraded":"2024-09-13T16:48:22.342883+0000","last_fullsized":"2024-09-13T16:48:22.342883+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'13","reported_seq":27,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:46:13.069441+0000","last_change":"2024-09-13T16:43:54.559551+0000","last_active":"2024-09-13T16:46:13.069441+0000","last_peered":"2024-09-13T16:46:13.069441+0000","last_clean":"2024-09-13T16:46:13.069441+0000","last_became_active":"2024-09-13T16:43:54.559290+0000","last_became_peered":"2024-09-13T16:43:54.559290+0000","last_unstale":"2024-09-13T16:46:13.069441+0000","last_undegraded":"2024-09-13T16:46:13.069441+0000","last_fullsized":"2024-09-13T16:46:13.069441+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:07.279335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"22'180","reported_seq":195,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.351450+0000","last_change":"2024-09-13T16:43:55.000783+0000","last_active":"2024-09-13T16:48:22.351450+0000","last_peered":"2024-09-13T16:48:22.351450+0000","last_clean":"2024-09-13T16:48:22.351450+0000","last_became_active":"2024-09-13T16:43:55.000196+0000","last_became_peered":"2024-09-13T16:43:55.000196+0000","last_unstale":"2024-09-13T16:48:22.351450+0000","last_undegraded":"2024-09-13T16:48:22.351450+0000","last_fullsized":"2024-09-13T16:48:22.351450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:47:54.560039+0000","last_change":"2024-09-13T16:43:54.559886+0000","last_active":"2024-09-13T16:47:54.560039+0000","last_peered":"2024-09-13T16:47:54.560039+0000","last_clean":"2024-09-13T16:47:54.560039+0000","last_became_active":"2024-09-13T16:43:54.559586+0000","last_became_peered":"2024-09-13T16:43:54.559586+0000","last_unstale":"2024-09-13T16:47:54.560039+0000","last_undegraded":"2024-09-13T16:47:54.560039+0000","last_fullsized":"2024-09-13T16:47:54.560039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:53.515146+0000","last_clean_scrub_stamp":"2024-09-13T16:43:53.515146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"22'152","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.337807+0000","last_change":"2024-09-13T16:43:54.996099+0000","last_active":"2024-09-13T16:48:22.337807+0000","last_peered":"2024-09-13T16:48:22.337807+0000","last_clean":"2024-09-13T16:48:22.337807+0000","last_became_active":"2024-09-13T16:43:54.995802+0000","last_became_peered":"2024-09-13T16:43:54.995802+0000","last_unstale":"2024-09-13T16:48:22.337807+0000","last_undegraded":"2024-09-13T16:48:22.337807+0000","last_fullsized":"2024-09-13T16:48:22.337807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'19","reported_seq":46,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:27.429267+0000","last_change":"2024-09-13T16:48:27.429354+0000","last_active":"2024-09-13T16:48:27.429267+0000","last_peered":"2024-09-13T16:48:27.429267+0000","last_clean":"2024-09-13T16:48:27.429267+0000","last_became_active":"2024-09-13T16:43:54.560552+0000","last_became_peered":"2024-09-13T16:43:54.560552+0000","last_unstale":"2024-09-13T16:48:27.429267+0000","last_undegraded":"2024-09-13T16:48:27.429267+0000","last_fullsized":"2024-09-13T16:48:27.429267+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'19","last_scrub_stamp":"2024-09-13T16:48:27.429247+0000","last_deep_scrub":"22'19","last_deep_scrub_stamp":"2024-09-13T16:48:27.429247+0000","last_clean_scrub_stamp":"2024-09-13T16:48:27.429247+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:55:11.474511+0000","scrub_duration":0.0049610629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11253,"ondisk_log_size":11253,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":371833,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":410,"num_write_kb":567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":743666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":388,"ondisk_log_size":388,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39628,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332212,"statfs":{"total":96636764160,"available":96596185088,"internally_reserved":0,"allocated":331776,"data_stored":675500,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":40237514},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.59199999999999997,"15min":0.57299999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":5.0620000000000003,"5min":5.0620000000000003,"15min":5.0620000000000003},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.68500000000000005,"15min":0.60099999999999998},"min":{"1min":0.372,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":5.1859999999999999,"5min":5.1859999999999999,"15min":5.1859999999999999},"last":0.60599999999999998}]},{"osd":1,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.36799999999999999,"5min":0.314,"15min":0.314},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.61699999999999999,"15min":0.58999999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":3.6070000000000002,"5min":3.6070000000000002,"15min":3.6070000000000002},"last":0.58799999999999997}]},{"osd":2,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":2.4929999999999999,"5min":2.4929999999999999,"15min":2.4929999999999999},"last":0.62},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.63400000000000001,"15min":0.61599999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":0.57399999999999995}]},{"osd":3,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.56899999999999995,"15min":0.58199999999999996},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.8490000000000002,"5min":3.8490000000000002,"15min":3.8490000000000002},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64800000000000002,"15min":0.60499999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.3160000000000001,"5min":4.0129999999999999,"15min":4.0129999999999999},"last":0.55700000000000005}]},{"osd":4,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.434,"15min":0.442},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.78000000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45000000000000001,"15min":0.49199999999999999},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67300000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.33100000000000002}]},{"osd":5,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46300000000000002,"15min":0.48699999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.499,"15min":0.52200000000000002},"min":{"1min":0.32600000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.753,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.372}]},{"osd":6,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.53300000000000003,"15min":0.56299999999999994},"min":{"1min":0.36399999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81100000000000005,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.504,"15min":0.54000000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72799999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.44600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35812,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336028,"statfs":{"total":96636764160,"available":96600092672,"internally_reserved":0,"allocated":618496,"data_stored":448920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36043209},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":0.69499999999999995,"15min":0.625},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":8.468,"5min":8.468,"15min":8.468},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.67300000000000004,"15min":0.625},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":7.5739999999999998,"5min":7.5739999999999998,"15min":7.5739999999999998},"last":0.65100000000000002}]},{"osd":1,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.29299999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60799999999999998,"15min":0.63900000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.71699999999999997}]},{"osd":2,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.028,"5min":2.431,"15min":2.431},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.466,"5min":0.78500000000000003,"15min":0.64100000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.9139999999999997,"5min":6.9139999999999997,"15min":6.9139999999999997},"last":0.753}]},{"osd":3,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":0.67200000000000004,"15min":0.60999999999999999},"min":{"1min":0.373,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.2400000000000002,"5min":6.2400000000000002,"15min":6.2400000000000002},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.70899999999999996,"15min":0.66200000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.4500000000000002,"5min":3.4500000000000002,"15min":3.4500000000000002},"last":0.73499999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.41299999999999998,"15min":0.45100000000000001},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.54300000000000004,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.439,"15min":0.48299999999999998},"min":{"1min":0.23699999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.38500000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.48799999999999999,"15min":0.52800000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.51100000000000001,"15min":0.56499999999999995},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.73599999999999999,"5min":0.752,"15min":0.752},"last":0.42199999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.48199999999999998,"15min":0.52000000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.71299999999999997,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.54000000000000004,"15min":0.58099999999999996},"min":{"1min":0.34000000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.44}]}]},{"osd":5,"up_from":13,"seq":55834574905,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39572,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332268,"statfs":{"total":96636764160,"available":96596242432,"internally_reserved":0,"allocated":274432,"data_stored":86302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.58299999999999996,"15min":0.56299999999999994},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.411,"5min":3.411,"15min":3.411},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.117,"5min":3.117,"15min":3.117},"last":0.63100000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55500000000000005,"15min":0.56100000000000005},"min":{"1min":0.29899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.64300000000000002,"15min":0.58899999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.843,"5min":4.3860000000000001,"15min":4.3860000000000001},"last":0.55800000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.67300000000000004,"15min":0.64300000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":4.0819999999999999,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.64700000000000002,"15min":0.61399999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":4.3170000000000002,"5min":4.3170000000000002,"15min":4.3170000000000002},"last":0.53900000000000003}]},{"osd":3,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.90700000000000003,"15min":0.68700000000000006},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.4690000000000003,"5min":28.707000000000001,"15min":28.707000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.68700000000000006,"15min":0.625},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.2599999999999998,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":0.61799999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.41899999999999998,"15min":0.45100000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.45500000000000002,"15min":0.48099999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.70399999999999996,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.58199999999999996}]},{"osd":6,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.441,"15min":0.45700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.67000000000000004,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.46500000000000002,"15min":0.48499999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.50800000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.49099999999999999,"15min":0.52000000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65600000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.501,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.64900000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39552,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332288,"statfs":{"total":96636764160,"available":96596262912,"internally_reserved":0,"allocated":253952,"data_stored":84762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":3.7160000000000002,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.61099999999999999,"15min":0.59399999999999997},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.448,"5min":2.9980000000000002,"15min":2.9980000000000002},"last":0.44900000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.69599999999999995,"15min":0.63},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.3700000000000001,"5min":4.3700000000000001,"15min":4.3700000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.63500000000000001,"15min":0.61499999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.9020000000000001,"5min":3.9020000000000001,"15min":3.9020000000000001},"last":0.433}]},{"osd":2,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":2.085,"5min":2.613,"15min":2.613},"last":0.55100000000000005},{"interface":"front","average":{"1min":1.194,"5min":0.72499999999999998,"15min":0.65800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":7.9809999999999999,"5min":7.9809999999999999,"15min":7.9809999999999999},"last":0.47399999999999998}]},{"osd":3,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.65800000000000003,"15min":0.63400000000000001},"min":{"1min":0.36199999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.68600000000000005,"15min":0.68899999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.8130000000000002,"5min":2.8130000000000002,"15min":2.8130000000000002},"last":0.49199999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.42799999999999999,"15min":0.46100000000000002},"min":{"1min":0.25600000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.371},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.438,"15min":0.47299999999999998},"min":{"1min":0.254,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61399999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.38100000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.442,"15min":0.47999999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.628,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.48599999999999999,"15min":0.53400000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.63600000000000001,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.33700000000000002}]},{"osd":7,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.48099999999999998,"15min":0.50900000000000001},"min":{"1min":0.308,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.54400000000000004,"15min":0.60399999999999998},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70499999999999996,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.41999999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44012,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94327828,"statfs":{"total":96636764160,"available":96591695872,"internally_reserved":0,"allocated":626688,"data_stored":449059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":44431816},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.57899999999999996,"15min":0.49199999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.8449999999999998,"5min":4.8449999999999998,"15min":4.8449999999999998},"last":0.316},{"interface":"front","average":{"1min":1.5529999999999999,"5min":0.77800000000000002,"15min":0.59799999999999998},"min":{"1min":0.41099999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":7.7489999999999997,"5min":7.7489999999999997,"15min":7.7489999999999997},"last":0.51400000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":2.0529999999999999,"5min":0.84299999999999997,"15min":0.60499999999999998},"min":{"1min":0.317,"5min":0.252,"15min":0.252},"max":{"1min":28.908999999999999,"5min":28.908999999999999,"15min":28.908999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.0920000000000001,"5min":0.73299999999999998,"15min":0.64600000000000002},"min":{"1min":0.40600000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.1520000000000001,"5min":3.1699999999999999,"15min":3.1699999999999999},"last":0.504}]},{"osd":2,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":0.80300000000000005,"15min":0.63500000000000001},"min":{"1min":0.41799999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":4.3630000000000004,"5min":4.3630000000000004,"15min":4.3630000000000004},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.76900000000000002,"15min":0.69299999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.1859999999999999,"5min":3.2610000000000001,"15min":3.2610000000000001},"last":0.44800000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":0.94299999999999995,"15min":0.82699999999999996},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":4.1029999999999998,"5min":4.1029999999999998,"15min":4.1029999999999998},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.79300000000000004,"15min":0.753},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.903,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":0.59799999999999998}]},{"osd":5,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":0.80700000000000005,"15min":0.754},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":2.012,"5min":2.5790000000000002,"15min":2.5790000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.1160000000000001,"5min":0.89100000000000001,"15min":0.84699999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.0739999999999998,"5min":3.1970000000000001,"15min":3.1970000000000001},"last":0.64300000000000002}]},{"osd":6,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":0.96699999999999997,"15min":0.84199999999999997},"min":{"1min":0.61099999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":4.1449999999999996,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.622},{"interface":"front","average":{"1min":1.0760000000000001,"5min":0.86499999999999999,"15min":0.82799999999999996},"min":{"1min":0.53000000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.9319999999999999,"5min":2.7360000000000002,"15min":2.7360000000000002},"last":0.72099999999999997}]},{"osd":7,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":0.88500000000000001,"15min":0.82399999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":3.117,"5min":3.2320000000000002,"15min":3.2320000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.613,"5min":1.006,"15min":0.874},"min":{"1min":0.56599999999999995,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":4.181,"5min":4.181,"15min":4.181},"last":0.65100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35464,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336376,"statfs":{"total":96636764160,"available":96600449024,"internally_reserved":0,"allocated":262144,"data_stored":84385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.988,"5min":0.996,"15min":0.65800000000000003},"min":{"1min":0.26600000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":31.440999999999999,"5min":31.440999999999999,"15min":31.440999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.294,"5min":0.69099999999999995,"15min":0.57699999999999996},"min":{"1min":0.34899999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.7720000000000002,"5min":4.7720000000000002,"15min":4.7720000000000002},"last":0.38200000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.2949999999999999,"5min":0.89800000000000002,"15min":0.65600000000000003},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.121,"5min":0.755,"15min":0.61299999999999999},"min":{"1min":0.379,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.0230000000000001,"5min":6.5659999999999998,"15min":6.5659999999999998},"last":0.46000000000000002}]},{"osd":3,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":0.71499999999999997,"15min":0.59299999999999997},"min":{"1min":0.371,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":6.5469999999999997,"5min":6.5469999999999997,"15min":6.5469999999999997},"last":0.443},{"interface":"front","average":{"1min":1.7869999999999999,"5min":0.84499999999999997,"15min":0.68300000000000005},"min":{"1min":0.45500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":8.8659999999999997,"5min":8.8659999999999997,"15min":8.8659999999999997},"last":0.53600000000000003}]},{"osd":4,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":0.81499999999999995,"15min":0.752},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":3.4460000000000002,"5min":3.4460000000000002,"15min":3.4460000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.163,"5min":0.80600000000000005,"15min":0.76300000000000001},"min":{"1min":0.46200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":6.4379999999999997,"5min":6.4379999999999997,"15min":6.4379999999999997},"last":0.70399999999999996}]},{"osd":5,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.929,"5min":0.97599999999999998,"15min":0.81699999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":9.1039999999999992,"5min":9.1039999999999992,"15min":9.1039999999999992},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":0.84399999999999997,"15min":0.77400000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":3.633,"5min":3.633,"15min":3.633},"last":0.66500000000000004}]},{"osd":6,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":0.83199999999999996,"15min":0.79400000000000004},"min":{"1min":0.54400000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.95,"5min":0.999,"15min":0.85899999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":9.141,"5min":9.141,"15min":9.141},"last":0.65100000000000002}]},{"osd":7,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":0.88400000000000001,"15min":0.84299999999999997},"min":{"1min":0.50700000000000001,"5min":0.505,"15min":0.505},"max":{"1min":3.5990000000000002,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.2450000000000001,"5min":0.86699999999999999,"15min":0.83499999999999996},"min":{"1min":0.57599999999999996,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":6.4800000000000004,"5min":6.4800000000000004,"15min":6.4800000000000004},"last":0.63900000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35460,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336380,"statfs":{"total":96636764160,"available":96600453120,"internally_reserved":0,"allocated":258048,"data_stored":84564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5630000000000002,"5min":1.0009999999999999,"15min":0.65200000000000002},"min":{"1min":0.29399999999999998,"5min":0.249,"15min":0.249},"max":{"1min":22.721,"5min":22.721,"15min":22.721},"last":0.33000000000000002},{"interface":"front","average":{"1min":1.226,"5min":0.86699999999999999,"15min":0.64100000000000001},"min":{"1min":0.32200000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.2430000000000003,"5min":19.494,"15min":19.494},"last":0.46800000000000003}]},{"osd":2,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3570000000000002,"5min":0.98999999999999999,"15min":0.67200000000000004},"min":{"1min":0.34799999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":15.491,"5min":15.491,"15min":15.491},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.5549999999999999,"5min":0.93300000000000005,"15min":0.70399999999999996},"min":{"1min":0.39600000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.6829999999999998,"5min":17.071000000000002,"15min":17.071000000000002},"last":0.48399999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.631,"5min":0.98599999999999999,"15min":0.73499999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":6.883,"5min":18.838000000000001,"15min":18.838000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.339,"5min":0.92200000000000004,"15min":0.69699999999999995},"min":{"1min":0.40899999999999997,"5min":0.31,"15min":0.31},"max":{"1min":7.0339999999999998,"5min":20.199000000000002,"15min":20.199000000000002},"last":0.42499999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.778,"5min":1.2390000000000001,"15min":0.91300000000000003},"min":{"1min":0.39800000000000002,"5min":0.373,"15min":0.373},"max":{"1min":15.654999999999999,"5min":15.654999999999999,"15min":15.654999999999999},"last":0.629},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.091,"15min":0.89500000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":7.5659999999999998,"5min":18.742000000000001,"15min":18.742000000000001},"last":0.68000000000000005}]},{"osd":5,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":0.97699999999999998,"15min":0.89300000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":7.3230000000000004,"5min":7.3230000000000004,"15min":7.3230000000000004},"last":0.65000000000000002},{"interface":"front","average":{"1min":2.819,"5min":1.296,"15min":0.98299999999999998},"min":{"1min":0.51400000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":15.686,"5min":15.686,"15min":15.686},"last":0.69999999999999996}]},{"osd":6,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":0.96799999999999997,"15min":0.86599999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":7.3700000000000001,"5min":7.3700000000000001,"15min":7.3700000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":2.851,"5min":1.3180000000000001,"15min":1.002},"min":{"1min":0.55500000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":15.718,"5min":15.718,"15min":15.718},"last":0.66600000000000004}]},{"osd":7,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":1.119,"15min":0.89500000000000002},"min":{"1min":0.495,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":8.1539999999999999,"5min":18.798999999999999,"15min":18.798999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.4339999999999999,"5min":0.995,"15min":0.90200000000000002},"min":{"1min":0.53000000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":7.407,"5min":7.4950000000000001,"15min":7.4950000000000001},"last":0.61399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574906,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39616,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332224,"statfs":{"total":96636764160,"available":96596197376,"internally_reserved":0,"allocated":319488,"data_stored":673303,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.65800000000000003,"15min":0.53000000000000003},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.9239999999999999,"5min":7.2539999999999996,"15min":7.2539999999999996},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.2589999999999999,"5min":0.71999999999999997,"15min":0.55700000000000005},"min":{"1min":0.32400000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":6.0949999999999998,"5min":6.0949999999999998,"15min":6.0949999999999998},"last":0.55400000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.78300000000000003,"15min":0.627},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.1800000000000002,"5min":7.1920000000000002,"15min":7.1920000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.151,"5min":0.74299999999999999,"15min":0.59899999999999998},"min":{"1min":0.36399999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":6.5,"5min":6.5,"15min":6.5},"last":0.50700000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":0.66500000000000004,"15min":0.54800000000000004},"min":{"1min":0.38800000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":4.4459999999999997,"5min":4.4459999999999997,"15min":4.4459999999999997},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.85999999999999999,"15min":0.69099999999999995},"min":{"1min":0.38700000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0409999999999999,"5min":10.522,"15min":10.522},"last":0.43099999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":0.89300000000000002,"15min":0.80100000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":6.4560000000000004,"5min":6.4560000000000004,"15min":6.4560000000000004},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.82799999999999996,"15min":0.77900000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":3.79,"5min":3.79,"15min":3.79},"last":0.80600000000000005}]},{"osd":5,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":0.85399999999999998,"15min":0.78800000000000003},"min":{"1min":0.51300000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.8340000000000001,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.002,"5min":0.96599999999999997,"15min":0.85399999999999998},"min":{"1min":0.48599999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":4.5620000000000003,"5min":7.8339999999999996,"15min":7.8339999999999996},"last":0.64900000000000002}]},{"osd":6,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.98199999999999998,"15min":0.86499999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.7270000000000003,"5min":7.5510000000000002,"15min":7.5510000000000002},"last":0.624},{"interface":"front","average":{"1min":1.345,"5min":0.95699999999999996,"15min":0.81499999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":7.444,"5min":7.444,"15min":7.444},"last":0.82499999999999996}]},{"osd":7,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":0.95399999999999996,"15min":0.81599999999999995},"min":{"1min":0.51900000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":7.5350000000000001,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.1639999999999999,"5min":0.874,"15min":0.79700000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.1879999999999997,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.88700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":365979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":389120,"data_stored":365840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:48:30.401 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.402 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.402 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.402 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.402 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.402 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.402 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.402 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.402 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.402 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.403 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.403 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.403 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.403 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.403 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.403 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.403 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.403 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.404 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.404 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.404 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.404 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.404 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.404 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.404 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.404 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.405 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.405 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.405 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.405 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.405 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.405 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.405 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.405 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.405 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.406 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.406 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.406 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.406 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.406 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.406 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.406 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.406 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.406 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.407 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.407 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.407 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.407 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.407 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.407 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.407 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.407 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.408 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.408 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.408 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.408 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.408 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.408 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.408 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.408 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.409 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.409 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-13T16:43:50.498116+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=50, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.409 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.409 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.409 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.409 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.409 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.409 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.410 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.410 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.410 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.410 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.410 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.410 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.410 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.410 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.410 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.411 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.411 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.411 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.411 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.411 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.411 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.411 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.411 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.412 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.412 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.412 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.412 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.412 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.412 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.412 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-09-13T16:43:53.515146+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=53, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.412 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:30.413 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T16:48:34.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:48:34.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:48:50.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:48:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.583+0000 7fc463e00640 1 Processor -- start 2024-09-13T16:48:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.583+0000 7fc463e00640 1 -- start start 2024-09-13T16:48:50.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.584+0000 7fc463e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc4640c9b50 0x7fc4640c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:50.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.585+0000 7fc463e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc464152f70 0x7fc4640c56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:50.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.585+0000 7fc463e00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc464158a00 con 0x7fc4640c9f30 2024-09-13T16:48:50.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.585+0000 7fc463e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4640c5c20 con 0x7fc4640c9b50 2024-09-13T16:48:50.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.585+0000 7fc463e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc464158cc0 con 0x7fc464152f70 2024-09-13T16:48:50.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.585+0000 7fc462a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc4640c9b50 0x7fc4640c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:50.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.585+0000 7fc462a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc4640c9b50 0x7fc4640c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47536/0 (socket says 172.21.15.28:47536) 2024-09-13T16:48:50.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.585+0000 7fc462a00640 1 -- 172.21.15.28:0/3882954924 learned_addr learned my addr 172.21.15.28:0/3882954924 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:50.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.585+0000 7fc463400640 1 --2- 172.21.15.28:0/3882954924 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc464152f70 0x7fc4640c56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:50.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.585+0000 7fc462a00640 1 -- 172.21.15.28:0/3882954924 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc464152f70 msgr2=0x7fc4640c56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:48:50.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.585+0000 7fc462a00640 1 --2- 172.21.15.28:0/3882954924 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc464152f70 0x7fc4640c56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:50.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.585+0000 7fc462a00640 1 -- 172.21.15.28:0/3882954924 >> v1:172.21.15.191:6789/0 conn(0x7fc4640c9f30 legacy=0x7fc464152860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:50.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.585+0000 7fc462a00640 1 -- 172.21.15.28:0/3882954924 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc464158fa0 con 0x7fc4640c9b50 2024-09-13T16:48:50.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.585+0000 7fc461600640 1 -- 172.21.15.28:0/3882954924 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680826280 0 0) 0x7fc464158a00 con 0x7fc4640c9f30 2024-09-13T16:48:50.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.586+0000 7fc462a00640 1 --2- 172.21.15.28:0/3882954924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4640c9b50 0x7fc4640c90d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc454009870 tx=0x7fc45402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f4093c07461b73 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:50.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.587+0000 7fc461600640 1 -- 172.21.15.28:0/3882954924 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc454032440 con 0x7fc4640c9b50 2024-09-13T16:48:50.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.587+0000 7fc461600640 1 -- 172.21.15.28:0/3882954924 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4540325a0 con 0x7fc4640c9b50 2024-09-13T16:48:50.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.587+0000 7fc461600640 1 -- 172.21.15.28:0/3882954924 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc454032930 con 0x7fc4640c9b50 2024-09-13T16:48:50.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.587+0000 7fc463e00640 1 -- 172.21.15.28:0/3882954924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4640c9b50 msgr2=0x7fc4640c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:50.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.587+0000 7fc463e00640 1 --2- 172.21.15.28:0/3882954924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4640c9b50 0x7fc4640c90d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc454009870 tx=0x7fc45402a270 comp rx=0 tx=0).stop 2024-09-13T16:48:50.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.587+0000 7fc463e00640 1 -- 172.21.15.28:0/3882954924 shutdown_connections 2024-09-13T16:48:50.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.587+0000 7fc463e00640 1 --2- 172.21.15.28:0/3882954924 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc464152f70 0x7fc4640c56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:50.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.587+0000 7fc463e00640 1 --2- 172.21.15.28:0/3882954924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4640c9b50 0x7fc4640c90d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:50.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.587+0000 7fc463e00640 1 -- 172.21.15.28:0/3882954924 >> 172.21.15.28:0/3882954924 conn(0x7fc46406ca50 msgr2=0x7fc46406a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:50.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.587+0000 7fc463e00640 1 -- 172.21.15.28:0/3882954924 shutdown_connections 2024-09-13T16:48:50.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.587+0000 7fc463e00640 1 -- 172.21.15.28:0/3882954924 wait complete. 2024-09-13T16:48:50.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.589+0000 7fc463e00640 1 Processor -- start 2024-09-13T16:48:50.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.589+0000 7fc463e00640 1 -- start start 2024-09-13T16:48:50.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.590+0000 7fc463e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4640c9b50 0x7fc4641568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:50.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.590+0000 7fc463e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4640c9f30 0x7fc4640c2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:50.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.590+0000 7fc463e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc464152f70 0x7fc4640c29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:50.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.590+0000 7fc463e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4640c1f30 con 0x7fc4640c9f30 2024-09-13T16:48:50.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.590+0000 7fc463e00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4641eeb30 con 0x7fc4640c9b50 2024-09-13T16:48:50.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.590+0000 7fc463e00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4641eee10 con 0x7fc464152f70 2024-09-13T16:48:50.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.590+0000 7fc462000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4640c9f30 0x7fc4640c2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.590+0000 7fc462000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4640c9f30 0x7fc4640c2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47540/0 (socket says 172.21.15.28:47540) 2024-09-13T16:48:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.590+0000 7fc462000640 1 -- 172.21.15.28:0/2900277375 learned_addr learned my addr 172.21.15.28:0/2900277375 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:48:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.590+0000 7fc462a00640 1 --2- 172.21.15.28:0/2900277375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4640c9b50 0x7fc4641568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.590+0000 7fc463400640 1 --2- 172.21.15.28:0/2900277375 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc464152f70 0x7fc4640c29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.591+0000 7fc463400640 1 -- 172.21.15.28:0/2900277375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4640c9b50 msgr2=0x7fc4641568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.591+0000 7fc463400640 1 --2- 172.21.15.28:0/2900277375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4640c9b50 0x7fc4641568d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.591+0000 7fc463400640 1 -- 172.21.15.28:0/2900277375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4640c9f30 msgr2=0x7fc4640c2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.591+0000 7fc463400640 1 --2- 172.21.15.28:0/2900277375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4640c9f30 0x7fc4640c2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.591+0000 7fc463400640 1 -- 172.21.15.28:0/2900277375 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4540094d0 con 0x7fc464152f70 2024-09-13T16:48:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.591+0000 7fc462a00640 1 --2- 172.21.15.28:0/2900277375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4640c9b50 0x7fc4641568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:48:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.591+0000 7fc463400640 1 --2- 172.21.15.28:0/2900277375 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc464152f70 0x7fc4640c29b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc4640b3c20 tx=0x7fc45c00ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.591+0000 7fc45b400640 1 -- 172.21.15.28:0/2900277375 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c009c00 con 0x7fc464152f70 2024-09-13T16:48:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.591+0000 7fc45b400640 1 -- 172.21.15.28:0/2900277375 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45c00f040 con 0x7fc464152f70 2024-09-13T16:48:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.591+0000 7fc463e00640 1 -- 172.21.15.28:0/2900277375 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4640c2180 con 0x7fc464152f70 2024-09-13T16:48:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.591+0000 7fc45b400640 1 -- 172.21.15.28:0/2900277375 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c0146c0 con 0x7fc464152f70 2024-09-13T16:48:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.591+0000 7fc463e00640 1 -- 172.21.15.28:0/2900277375 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4640c3d00 con 0x7fc464152f70 2024-09-13T16:48:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.592+0000 7fc463e00640 1 -- 172.21.15.28:0/2900277375 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4640c3b60 con 0x7fc464152f70 2024-09-13T16:48:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.593+0000 7fc45b400640 1 -- 172.21.15.28:0/2900277375 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc45c014820 con 0x7fc464152f70 2024-09-13T16:48:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.594+0000 7fc45b400640 1 --2- 172.21.15.28:0/2900277375 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc44406ac00 0x7fc44406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:48:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.594+0000 7fc45b400640 1 -- 172.21.15.28:0/2900277375 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc45c08a0f0 con 0x7fc464152f70 2024-09-13T16:48:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.594+0000 7fc462a00640 1 --2- 172.21.15.28:0/2900277375 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc44406ac00 0x7fc44406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:48:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.595+0000 7fc462a00640 1 --2- 172.21.15.28:0/2900277375 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc44406ac00 0x7fc44406d0c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc454034000 tx=0x7fc45402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:48:50.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.599+0000 7fc45b400640 1 -- 172.21.15.28:0/2900277375 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc4640c3b60 con 0x7fc464152f70 2024-09-13T16:48:50.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.775+0000 7fc463e00640 1 -- 172.21.15.28:0/2900277375 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc4640c71c0 con 0x7fc44406ac00 2024-09-13T16:48:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.784+0000 7fc45b400640 1 -- 172.21.15.28:0/2900277375 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290234 (secure 0 0 0) 0x7fc4640c71c0 con 0x7fc44406ac00 2024-09-13T16:48:50.789 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:48:50.791 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T16:48:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.788+0000 7fc463e00640 1 -- 172.21.15.28:0/2900277375 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc44406ac00 msgr2=0x7fc44406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.788+0000 7fc463e00640 1 --2- 172.21.15.28:0/2900277375 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc44406ac00 0x7fc44406d0c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc454034000 tx=0x7fc45402e000 comp rx=0 tx=0).stop 2024-09-13T16:48:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.788+0000 7fc463e00640 1 -- 172.21.15.28:0/2900277375 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc464152f70 msgr2=0x7fc4640c29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.788+0000 7fc463e00640 1 --2- 172.21.15.28:0/2900277375 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc464152f70 0x7fc4640c29b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc4640b3c20 tx=0x7fc45c00ef90 comp rx=0 tx=0).stop 2024-09-13T16:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.788+0000 7fc463e00640 1 -- 172.21.15.28:0/2900277375 shutdown_connections 2024-09-13T16:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.788+0000 7fc463e00640 1 --2- 172.21.15.28:0/2900277375 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc44406ac00 0x7fc44406d0c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.788+0000 7fc463e00640 1 --2- 172.21.15.28:0/2900277375 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc464152f70 0x7fc4640c29b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.788+0000 7fc463e00640 1 --2- 172.21.15.28:0/2900277375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4640c9f30 0x7fc4640c2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:50.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.788+0000 7fc463e00640 1 --2- 172.21.15.28:0/2900277375 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4640c9b50 0x7fc4641568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.788+0000 7fc463e00640 1 -- 172.21.15.28:0/2900277375 >> 172.21.15.28:0/2900277375 conn(0x7fc46406ca50 msgr2=0x7fc4640ca330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.788+0000 7fc463e00640 1 -- 172.21.15.28:0/2900277375 shutdown_connections 2024-09-13T16:48:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:48:50.788+0000 7fc463e00640 1 -- 172.21.15.28:0/2900277375 wait complete. 2024-09-13T16:48:50.806 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-09-13T16:48:49.659420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":962201,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":121,"num_read_kb":105,"num_write":9785,"num_write_kb":25168807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11837,"ondisk_log_size":11837,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":313212,"kb_used_data":2876,"kb_used_omap":12,"kb_used_meta":310259,"kb_avail":754661508,"statfs":{"total":773094113280,"available":772773384192,"internally_reserved":0,"allocated":2945024,"data_stored":2586795,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12743,"internal_metadata":317705785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15000,"num_omap_keys":30,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002922"},"pg_stats":[{"pgid":"3.3f","version":"22'187","reported_seq":202,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.335066+0000","last_change":"2024-09-13T16:43:54.987241+0000","last_active":"2024-09-13T16:48:22.335066+0000","last_peered":"2024-09-13T16:48:22.335066+0000","last_clean":"2024-09-13T16:48:22.335066+0000","last_became_active":"2024-09-13T16:43:54.986988+0000","last_became_peered":"2024-09-13T16:43:54.986988+0000","last_unstale":"2024-09-13T16:48:22.335066+0000","last_undegraded":"2024-09-13T16:48:22.335066+0000","last_fullsized":"2024-09-13T16:48:22.335066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"22'138","reported_seq":153,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.343644+0000","last_change":"2024-09-13T16:43:55.001141+0000","last_active":"2024-09-13T16:48:22.343644+0000","last_peered":"2024-09-13T16:48:22.343644+0000","last_clean":"2024-09-13T16:48:22.343644+0000","last_became_active":"2024-09-13T16:43:55.000757+0000","last_became_peered":"2024-09-13T16:43:55.000757+0000","last_unstale":"2024-09-13T16:48:22.343644+0000","last_undegraded":"2024-09-13T16:48:22.343644+0000","last_fullsized":"2024-09-13T16:48:22.343644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"22'212","reported_seq":238,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:35.441355+0000","last_change":"2024-09-13T16:48:35.441475+0000","last_active":"2024-09-13T16:48:35.441355+0000","last_peered":"2024-09-13T16:48:35.441355+0000","last_clean":"2024-09-13T16:48:35.441355+0000","last_became_active":"2024-09-13T16:43:54.986198+0000","last_became_peered":"2024-09-13T16:43:54.986198+0000","last_unstale":"2024-09-13T16:48:35.441355+0000","last_undegraded":"2024-09-13T16:48:35.441355+0000","last_fullsized":"2024-09-13T16:48:35.441355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'212","last_scrub_stamp":"2024-09-13T16:48:35.441330+0000","last_deep_scrub":"22'212","last_deep_scrub_stamp":"2024-09-13T16:48:35.441330+0000","last_clean_scrub_stamp":"2024-09-13T16:48:35.441330+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:44:17.520802+0000","scrub_duration":0.0017448349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"22'133","reported_seq":148,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.349361+0000","last_change":"2024-09-13T16:43:54.992249+0000","last_active":"2024-09-13T16:48:22.349361+0000","last_peered":"2024-09-13T16:48:22.349361+0000","last_clean":"2024-09-13T16:48:22.349361+0000","last_became_active":"2024-09-13T16:43:54.991804+0000","last_became_peered":"2024-09-13T16:43:54.991804+0000","last_unstale":"2024-09-13T16:48:22.349361+0000","last_undegraded":"2024-09-13T16:48:22.349361+0000","last_fullsized":"2024-09-13T16:48:22.349361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":270336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"22'194","reported_seq":208,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.339383+0000","last_change":"2024-09-13T16:43:54.987564+0000","last_active":"2024-09-13T16:48:22.339383+0000","last_peered":"2024-09-13T16:48:22.339383+0000","last_clean":"2024-09-13T16:48:22.339383+0000","last_became_active":"2024-09-13T16:43:54.987328+0000","last_became_peered":"2024-09-13T16:43:54.987328+0000","last_unstale":"2024-09-13T16:48:22.339383+0000","last_undegraded":"2024-09-13T16:48:22.339383+0000","last_fullsized":"2024-09-13T16:48:22.339383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"22'156","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.353586+0000","last_change":"2024-09-13T16:43:55.000019+0000","last_active":"2024-09-13T16:48:22.353586+0000","last_peered":"2024-09-13T16:48:22.353586+0000","last_clean":"2024-09-13T16:48:22.353586+0000","last_became_active":"2024-09-13T16:43:54.999752+0000","last_became_peered":"2024-09-13T16:43:54.999752+0000","last_unstale":"2024-09-13T16:48:22.353586+0000","last_undegraded":"2024-09-13T16:48:22.353586+0000","last_fullsized":"2024-09-13T16:48:22.353586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"22'175","reported_seq":189,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.339425+0000","last_change":"2024-09-13T16:43:54.999533+0000","last_active":"2024-09-13T16:48:22.339425+0000","last_peered":"2024-09-13T16:48:22.339425+0000","last_clean":"2024-09-13T16:48:22.339425+0000","last_became_active":"2024-09-13T16:43:54.999350+0000","last_became_peered":"2024-09-13T16:43:54.999350+0000","last_unstale":"2024-09-13T16:48:22.339425+0000","last_undegraded":"2024-09-13T16:48:22.339425+0000","last_fullsized":"2024-09-13T16:48:22.339425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"22'165","reported_seq":179,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.351084+0000","last_change":"2024-09-13T16:43:54.998526+0000","last_active":"2024-09-13T16:48:22.351084+0000","last_peered":"2024-09-13T16:48:22.351084+0000","last_clean":"2024-09-13T16:48:22.351084+0000","last_became_active":"2024-09-13T16:43:54.996309+0000","last_became_peered":"2024-09-13T16:43:54.996309+0000","last_unstale":"2024-09-13T16:48:22.351084+0000","last_undegraded":"2024-09-13T16:48:22.351084+0000","last_fullsized":"2024-09-13T16:48:22.351084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"22'155","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:29.997938+0000","last_change":"2024-09-13T16:43:54.993786+0000","last_active":"2024-09-13T16:48:29.997938+0000","last_peered":"2024-09-13T16:48:29.997938+0000","last_clean":"2024-09-13T16:48:29.997938+0000","last_became_active":"2024-09-13T16:43:54.993475+0000","last_became_peered":"2024-09-13T16:43:54.993475+0000","last_unstale":"2024-09-13T16:48:29.997938+0000","last_undegraded":"2024-09-13T16:48:29.997938+0000","last_fullsized":"2024-09-13T16:48:29.997938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"22'189","reported_seq":204,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:29.426116+0000","last_change":"2024-09-13T16:43:54.986272+0000","last_active":"2024-09-13T16:48:29.426116+0000","last_peered":"2024-09-13T16:48:29.426116+0000","last_clean":"2024-09-13T16:48:29.426116+0000","last_became_active":"2024-09-13T16:43:54.985979+0000","last_became_peered":"2024-09-13T16:43:54.985979+0000","last_unstale":"2024-09-13T16:48:29.426116+0000","last_undegraded":"2024-09-13T16:48:29.426116+0000","last_fullsized":"2024-09-13T16:48:29.426116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"22'174","reported_seq":200,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:42.717053+0000","last_change":"2024-09-13T16:48:42.717053+0000","last_active":"2024-09-13T16:48:42.717053+0000","last_peered":"2024-09-13T16:48:42.717053+0000","last_clean":"2024-09-13T16:48:42.717053+0000","last_became_active":"2024-09-13T16:43:54.985948+0000","last_became_peered":"2024-09-13T16:43:54.985948+0000","last_unstale":"2024-09-13T16:48:42.717053+0000","last_undegraded":"2024-09-13T16:48:42.717053+0000","last_fullsized":"2024-09-13T16:48:42.717053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'174","last_scrub_stamp":"2024-09-13T16:48:42.716932+0000","last_deep_scrub":"22'174","last_deep_scrub_stamp":"2024-09-13T16:48:42.716932+0000","last_clean_scrub_stamp":"2024-09-13T16:48:42.716932+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:32.131509+0000","scrub_duration":0.0014958700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"22'183","reported_seq":209,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:35.636447+0000","last_change":"2024-09-13T16:48:35.636447+0000","last_active":"2024-09-13T16:48:35.636447+0000","last_peered":"2024-09-13T16:48:35.636447+0000","last_clean":"2024-09-13T16:48:35.636447+0000","last_became_active":"2024-09-13T16:43:54.986161+0000","last_became_peered":"2024-09-13T16:43:54.986161+0000","last_unstale":"2024-09-13T16:48:35.636447+0000","last_undegraded":"2024-09-13T16:48:35.636447+0000","last_fullsized":"2024-09-13T16:48:35.636447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'183","last_scrub_stamp":"2024-09-13T16:48:35.636320+0000","last_deep_scrub":"22'183","last_deep_scrub_stamp":"2024-09-13T16:48:35.636320+0000","last_clean_scrub_stamp":"2024-09-13T16:48:35.636320+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:19.361757+0000","scrub_duration":0.00160545,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"22'170","reported_seq":185,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:27.707811+0000","last_change":"2024-09-13T16:43:54.997783+0000","last_active":"2024-09-13T16:48:27.707811+0000","last_peered":"2024-09-13T16:48:27.707811+0000","last_clean":"2024-09-13T16:48:27.707811+0000","last_became_active":"2024-09-13T16:43:54.997162+0000","last_became_peered":"2024-09-13T16:43:54.997162+0000","last_unstale":"2024-09-13T16:48:27.707811+0000","last_undegraded":"2024-09-13T16:48:27.707811+0000","last_fullsized":"2024-09-13T16:48:27.707811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"22'180","reported_seq":206,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:38.508357+0000","last_change":"2024-09-13T16:48:38.508357+0000","last_active":"2024-09-13T16:48:38.508357+0000","last_peered":"2024-09-13T16:48:38.508357+0000","last_clean":"2024-09-13T16:48:38.508357+0000","last_became_active":"2024-09-13T16:43:54.994891+0000","last_became_peered":"2024-09-13T16:43:54.994891+0000","last_unstale":"2024-09-13T16:48:38.508357+0000","last_undegraded":"2024-09-13T16:48:38.508357+0000","last_fullsized":"2024-09-13T16:48:38.508357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'180","last_scrub_stamp":"2024-09-13T16:48:38.508228+0000","last_deep_scrub":"22'180","last_deep_scrub_stamp":"2024-09-13T16:48:38.508228+0000","last_clean_scrub_stamp":"2024-09-13T16:48:38.508228+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:27.922805+0000","scrub_duration":0.0019548199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"22'192","reported_seq":218,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:46.055489+0000","last_change":"2024-09-13T16:48:46.055589+0000","last_active":"2024-09-13T16:48:46.055489+0000","last_peered":"2024-09-13T16:48:46.055489+0000","last_clean":"2024-09-13T16:48:46.055489+0000","last_became_active":"2024-09-13T16:43:54.981191+0000","last_became_peered":"2024-09-13T16:43:54.981191+0000","last_unstale":"2024-09-13T16:48:46.055489+0000","last_undegraded":"2024-09-13T16:48:46.055489+0000","last_fullsized":"2024-09-13T16:48:46.055489+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'192","last_scrub_stamp":"2024-09-13T16:48:46.055465+0000","last_deep_scrub":"22'192","last_deep_scrub_stamp":"2024-09-13T16:48:46.055465+0000","last_clean_scrub_stamp":"2024-09-13T16:48:46.055465+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:02:47.540947+0000","scrub_duration":0.0026106329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"22'160","reported_seq":175,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:28.855362+0000","last_change":"2024-09-13T16:43:54.994157+0000","last_active":"2024-09-13T16:48:28.855362+0000","last_peered":"2024-09-13T16:48:28.855362+0000","last_clean":"2024-09-13T16:48:28.855362+0000","last_became_active":"2024-09-13T16:43:54.993915+0000","last_became_peered":"2024-09-13T16:43:54.993915+0000","last_unstale":"2024-09-13T16:48:28.855362+0000","last_undegraded":"2024-09-13T16:48:28.855362+0000","last_fullsized":"2024-09-13T16:48:28.855362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"22'175","reported_seq":190,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:28.855368+0000","last_change":"2024-09-13T16:43:54.987328+0000","last_active":"2024-09-13T16:48:28.855368+0000","last_peered":"2024-09-13T16:48:28.855368+0000","last_clean":"2024-09-13T16:48:28.855368+0000","last_became_active":"2024-09-13T16:43:54.987048+0000","last_became_peered":"2024-09-13T16:43:54.987048+0000","last_unstale":"2024-09-13T16:48:28.855368+0000","last_undegraded":"2024-09-13T16:48:28.855368+0000","last_fullsized":"2024-09-13T16:48:28.855368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"22'142","reported_seq":168,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:41.453063+0000","last_change":"2024-09-13T16:48:41.453200+0000","last_active":"2024-09-13T16:48:41.453063+0000","last_peered":"2024-09-13T16:48:41.453063+0000","last_clean":"2024-09-13T16:48:41.453063+0000","last_became_active":"2024-09-13T16:43:54.985378+0000","last_became_peered":"2024-09-13T16:43:54.985378+0000","last_unstale":"2024-09-13T16:48:41.453063+0000","last_undegraded":"2024-09-13T16:48:41.453063+0000","last_fullsized":"2024-09-13T16:48:41.453063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'142","last_scrub_stamp":"2024-09-13T16:48:41.453040+0000","last_deep_scrub":"22'142","last_deep_scrub_stamp":"2024-09-13T16:48:41.453040+0000","last_clean_scrub_stamp":"2024-09-13T16:48:41.453040+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:30.867617+0000","scrub_duration":0.0021012510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"22'157","reported_seq":183,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:34.019076+0000","last_change":"2024-09-13T16:48:34.019175+0000","last_active":"2024-09-13T16:48:34.019076+0000","last_peered":"2024-09-13T16:48:34.019076+0000","last_clean":"2024-09-13T16:48:34.019076+0000","last_became_active":"2024-09-13T16:43:54.996523+0000","last_became_peered":"2024-09-13T16:43:54.996523+0000","last_unstale":"2024-09-13T16:48:34.019076+0000","last_undegraded":"2024-09-13T16:48:34.019076+0000","last_fullsized":"2024-09-13T16:48:34.019076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'157","last_scrub_stamp":"2024-09-13T16:48:34.019052+0000","last_deep_scrub":"22'157","last_deep_scrub_stamp":"2024-09-13T16:48:34.019052+0000","last_clean_scrub_stamp":"2024-09-13T16:48:34.019052+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:10.737127+0000","scrub_duration":0.0021886039999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"22'167","reported_seq":182,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:29.426083+0000","last_change":"2024-09-13T16:43:54.984879+0000","last_active":"2024-09-13T16:48:29.426083+0000","last_peered":"2024-09-13T16:48:29.426083+0000","last_clean":"2024-09-13T16:48:29.426083+0000","last_became_active":"2024-09-13T16:43:54.984595+0000","last_became_peered":"2024-09-13T16:43:54.984595+0000","last_unstale":"2024-09-13T16:48:29.426083+0000","last_undegraded":"2024-09-13T16:48:29.426083+0000","last_fullsized":"2024-09-13T16:48:29.426083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"22'167","reported_seq":193,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:48.554656+0000","last_change":"2024-09-13T16:48:48.554774+0000","last_active":"2024-09-13T16:48:48.554656+0000","last_peered":"2024-09-13T16:48:48.554656+0000","last_clean":"2024-09-13T16:48:48.554656+0000","last_became_active":"2024-09-13T16:43:54.996275+0000","last_became_peered":"2024-09-13T16:43:54.996275+0000","last_unstale":"2024-09-13T16:48:48.554656+0000","last_undegraded":"2024-09-13T16:48:48.554656+0000","last_fullsized":"2024-09-13T16:48:48.554656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'167","last_scrub_stamp":"2024-09-13T16:48:48.554633+0000","last_deep_scrub":"22'167","last_deep_scrub_stamp":"2024-09-13T16:48:48.554633+0000","last_clean_scrub_stamp":"2024-09-13T16:48:48.554633+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:08:39.168750+0000","scrub_duration":0.0023296100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"22'199","reported_seq":214,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:25.519017+0000","last_change":"2024-09-13T16:43:54.985368+0000","last_active":"2024-09-13T16:48:25.519017+0000","last_peered":"2024-09-13T16:48:25.519017+0000","last_clean":"2024-09-13T16:48:25.519017+0000","last_became_active":"2024-09-13T16:43:54.985130+0000","last_became_peered":"2024-09-13T16:43:54.985130+0000","last_unstale":"2024-09-13T16:48:25.519017+0000","last_undegraded":"2024-09-13T16:48:25.519017+0000","last_fullsized":"2024-09-13T16:48:25.519017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"22'158","reported_seq":173,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:27.707752+0000","last_change":"2024-09-13T16:43:54.991862+0000","last_active":"2024-09-13T16:48:27.707752+0000","last_peered":"2024-09-13T16:48:27.707752+0000","last_clean":"2024-09-13T16:48:27.707752+0000","last_became_active":"2024-09-13T16:43:54.991554+0000","last_became_peered":"2024-09-13T16:43:54.991554+0000","last_unstale":"2024-09-13T16:48:27.707752+0000","last_undegraded":"2024-09-13T16:48:27.707752+0000","last_fullsized":"2024-09-13T16:48:27.707752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"22'162","reported_seq":185,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:46.528414+0000","last_change":"2024-09-13T16:48:46.528520+0000","last_active":"2024-09-13T16:48:46.528414+0000","last_peered":"2024-09-13T16:48:46.528414+0000","last_clean":"2024-09-13T16:48:46.528414+0000","last_became_active":"2024-09-13T16:44:59.220762+0000","last_became_peered":"2024-09-13T16:44:59.220762+0000","last_unstale":"2024-09-13T16:48:46.528414+0000","last_undegraded":"2024-09-13T16:48:46.528414+0000","last_fullsized":"2024-09-13T16:48:46.528414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'162","last_scrub_stamp":"2024-09-13T16:48:46.528394+0000","last_deep_scrub":"22'162","last_deep_scrub_stamp":"2024-09-13T16:48:46.528394+0000","last_clean_scrub_stamp":"2024-09-13T16:48:46.528394+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:36:53.838527+0000","scrub_duration":0.0017451000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"22'170","reported_seq":196,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:46.934429+0000","last_change":"2024-09-13T16:48:46.934624+0000","last_active":"2024-09-13T16:48:46.934429+0000","last_peered":"2024-09-13T16:48:46.934429+0000","last_clean":"2024-09-13T16:48:46.934429+0000","last_became_active":"2024-09-13T16:43:54.991127+0000","last_became_peered":"2024-09-13T16:43:54.991127+0000","last_unstale":"2024-09-13T16:48:46.934429+0000","last_undegraded":"2024-09-13T16:48:46.934429+0000","last_fullsized":"2024-09-13T16:48:46.934429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'170","last_scrub_stamp":"2024-09-13T16:48:46.934406+0000","last_deep_scrub":"22'170","last_deep_scrub_stamp":"2024-09-13T16:48:46.934406+0000","last_clean_scrub_stamp":"2024-09-13T16:48:46.934406+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:08:37.548523+0000","scrub_duration":0.0024823760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"22'180","reported_seq":206,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:47.738029+0000","last_change":"2024-09-13T16:48:47.738136+0000","last_active":"2024-09-13T16:48:47.738029+0000","last_peered":"2024-09-13T16:48:47.738029+0000","last_clean":"2024-09-13T16:48:47.738029+0000","last_became_active":"2024-09-13T16:43:54.985299+0000","last_became_peered":"2024-09-13T16:43:54.985299+0000","last_unstale":"2024-09-13T16:48:47.738029+0000","last_undegraded":"2024-09-13T16:48:47.738029+0000","last_fullsized":"2024-09-13T16:48:47.738029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'180","last_scrub_stamp":"2024-09-13T16:48:47.738005+0000","last_deep_scrub":"22'180","last_deep_scrub_stamp":"2024-09-13T16:48:47.738005+0000","last_clean_scrub_stamp":"2024-09-13T16:48:47.738005+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:36:55.048139+0000","scrub_duration":0.001558622,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"22'147","reported_seq":162,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.342395+0000","last_change":"2024-09-13T16:43:54.993325+0000","last_active":"2024-09-13T16:48:22.342395+0000","last_peered":"2024-09-13T16:48:22.342395+0000","last_clean":"2024-09-13T16:48:22.342395+0000","last_became_active":"2024-09-13T16:43:54.993009+0000","last_became_peered":"2024-09-13T16:43:54.993009+0000","last_unstale":"2024-09-13T16:48:22.342395+0000","last_undegraded":"2024-09-13T16:48:22.342395+0000","last_fullsized":"2024-09-13T16:48:22.342395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"22'188","reported_seq":214,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:36.902401+0000","last_change":"2024-09-13T16:48:36.902509+0000","last_active":"2024-09-13T16:48:36.902401+0000","last_peered":"2024-09-13T16:48:36.902401+0000","last_clean":"2024-09-13T16:48:36.902401+0000","last_became_active":"2024-09-13T16:43:54.995774+0000","last_became_peered":"2024-09-13T16:43:54.995774+0000","last_unstale":"2024-09-13T16:48:36.902401+0000","last_undegraded":"2024-09-13T16:48:36.902401+0000","last_fullsized":"2024-09-13T16:48:36.902401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'188","last_scrub_stamp":"2024-09-13T16:48:36.902377+0000","last_deep_scrub":"22'188","last_deep_scrub_stamp":"2024-09-13T16:48:36.902377+0000","last_clean_scrub_stamp":"2024-09-13T16:48:36.902377+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:26.316954+0000","scrub_duration":0.0025868390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"22'172","reported_seq":198,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:45.453576+0000","last_change":"2024-09-13T16:48:45.453673+0000","last_active":"2024-09-13T16:48:45.453576+0000","last_peered":"2024-09-13T16:48:45.453576+0000","last_clean":"2024-09-13T16:48:45.453576+0000","last_became_active":"2024-09-13T16:43:55.001713+0000","last_became_peered":"2024-09-13T16:43:55.001713+0000","last_unstale":"2024-09-13T16:48:45.453576+0000","last_undegraded":"2024-09-13T16:48:45.453576+0000","last_fullsized":"2024-09-13T16:48:45.453576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'172","last_scrub_stamp":"2024-09-13T16:48:45.453552+0000","last_deep_scrub":"22'172","last_deep_scrub_stamp":"2024-09-13T16:48:45.453552+0000","last_clean_scrub_stamp":"2024-09-13T16:48:45.453552+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:24.544274+0000","scrub_duration":0.0017292360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"22'156","reported_seq":171,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.351091+0000","last_change":"2024-09-13T16:43:54.992745+0000","last_active":"2024-09-13T16:48:22.351091+0000","last_peered":"2024-09-13T16:48:22.351091+0000","last_clean":"2024-09-13T16:48:22.351091+0000","last_became_active":"2024-09-13T16:43:54.992508+0000","last_became_peered":"2024-09-13T16:43:54.992508+0000","last_unstale":"2024-09-13T16:48:22.351091+0000","last_undegraded":"2024-09-13T16:48:22.351091+0000","last_fullsized":"2024-09-13T16:48:22.351091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"22'143","reported_seq":169,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:40.496590+0000","last_change":"2024-09-13T16:48:40.496687+0000","last_active":"2024-09-13T16:48:40.496590+0000","last_peered":"2024-09-13T16:48:40.496590+0000","last_clean":"2024-09-13T16:48:40.496590+0000","last_became_active":"2024-09-13T16:43:54.986277+0000","last_became_peered":"2024-09-13T16:43:54.986277+0000","last_unstale":"2024-09-13T16:48:40.496590+0000","last_undegraded":"2024-09-13T16:48:40.496590+0000","last_fullsized":"2024-09-13T16:48:40.496590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'143","last_scrub_stamp":"2024-09-13T16:48:40.496565+0000","last_deep_scrub":"22'143","last_deep_scrub_stamp":"2024-09-13T16:48:40.496565+0000","last_clean_scrub_stamp":"2024-09-13T16:48:40.496565+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:29.395789+0000","scrub_duration":0.0022908500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"22'184","reported_seq":199,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.343604+0000","last_change":"2024-09-13T16:43:55.000059+0000","last_active":"2024-09-13T16:48:22.343604+0000","last_peered":"2024-09-13T16:48:22.343604+0000","last_clean":"2024-09-13T16:48:22.343604+0000","last_became_active":"2024-09-13T16:43:54.999778+0000","last_became_peered":"2024-09-13T16:43:54.999778+0000","last_unstale":"2024-09-13T16:48:22.343604+0000","last_undegraded":"2024-09-13T16:48:22.343604+0000","last_fullsized":"2024-09-13T16:48:22.343604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"22'219","reported_seq":244,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:32.569896+0000","last_change":"2024-09-13T16:48:32.570000+0000","last_active":"2024-09-13T16:48:32.569896+0000","last_peered":"2024-09-13T16:48:32.569896+0000","last_clean":"2024-09-13T16:48:32.569896+0000","last_became_active":"2024-09-13T16:43:54.988131+0000","last_became_peered":"2024-09-13T16:43:54.988131+0000","last_unstale":"2024-09-13T16:48:32.569896+0000","last_undegraded":"2024-09-13T16:48:32.569896+0000","last_fullsized":"2024-09-13T16:48:32.569896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'219","last_scrub_stamp":"2024-09-13T16:48:32.569873+0000","last_deep_scrub":"22'219","last_deep_scrub_stamp":"2024-09-13T16:48:32.569873+0000","last_clean_scrub_stamp":"2024-09-13T16:48:32.569873+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:13.799800+0000","scrub_duration":0.0023362790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'7","reported_seq":33,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:34.875848+0000","last_change":"2024-09-13T16:48:34.875982+0000","last_active":"2024-09-13T16:48:34.875848+0000","last_peered":"2024-09-13T16:48:34.875848+0000","last_clean":"2024-09-13T16:48:34.875848+0000","last_became_active":"2024-09-13T16:43:54.554697+0000","last_became_peered":"2024-09-13T16:43:54.554697+0000","last_unstale":"2024-09-13T16:48:34.875848+0000","last_undegraded":"2024-09-13T16:48:34.875848+0000","last_fullsized":"2024-09-13T16:48:34.875848+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'7","last_scrub_stamp":"2024-09-13T16:48:34.875828+0000","last_deep_scrub":"22'7","last_deep_scrub_stamp":"2024-09-13T16:48:34.875828+0000","last_clean_scrub_stamp":"2024-09-13T16:48:34.875828+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:12.773199+0000","scrub_duration":0.0042325979999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"22'173","reported_seq":198,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:43.449225+0000","last_change":"2024-09-13T16:48:43.449335+0000","last_active":"2024-09-13T16:48:43.449225+0000","last_peered":"2024-09-13T16:48:43.449225+0000","last_clean":"2024-09-13T16:48:43.449225+0000","last_became_active":"2024-09-13T16:43:54.998399+0000","last_became_peered":"2024-09-13T16:43:54.998399+0000","last_unstale":"2024-09-13T16:48:43.449225+0000","last_undegraded":"2024-09-13T16:48:43.449225+0000","last_fullsized":"2024-09-13T16:48:43.449225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'173","last_scrub_stamp":"2024-09-13T16:48:43.449202+0000","last_deep_scrub":"22'173","last_deep_scrub_stamp":"2024-09-13T16:48:43.449202+0000","last_clean_scrub_stamp":"2024-09-13T16:48:43.449202+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:41.461235+0000","scrub_duration":0.0019285179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'10","reported_seq":36,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:31.822652+0000","last_change":"2024-09-13T16:48:31.822764+0000","last_active":"2024-09-13T16:48:31.822652+0000","last_peered":"2024-09-13T16:48:31.822652+0000","last_clean":"2024-09-13T16:48:31.822652+0000","last_became_active":"2024-09-13T16:43:54.552977+0000","last_became_peered":"2024-09-13T16:43:54.552977+0000","last_unstale":"2024-09-13T16:48:31.822652+0000","last_undegraded":"2024-09-13T16:48:31.822652+0000","last_fullsized":"2024-09-13T16:48:31.822652+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'10","last_scrub_stamp":"2024-09-13T16:48:31.822629+0000","last_deep_scrub":"22'10","last_deep_scrub_stamp":"2024-09-13T16:48:31.822629+0000","last_clean_scrub_stamp":"2024-09-13T16:48:31.822629+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:13.052556+0000","scrub_duration":0.0055532949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"22'5","reported_seq":32,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:28.811089+0000","last_change":"2024-09-13T16:48:28.811089+0000","last_active":"2024-09-13T16:48:28.811089+0000","last_peered":"2024-09-13T16:48:28.811089+0000","last_clean":"2024-09-13T16:48:28.811089+0000","last_became_active":"2024-09-13T16:43:54.557722+0000","last_became_peered":"2024-09-13T16:43:54.557722+0000","last_unstale":"2024-09-13T16:48:28.811089+0000","last_undegraded":"2024-09-13T16:48:28.811089+0000","last_fullsized":"2024-09-13T16:48:28.811089+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'5","last_scrub_stamp":"2024-09-13T16:48:28.810988+0000","last_deep_scrub":"22'5","last_deep_scrub_stamp":"2024-09-13T16:48:28.810988+0000","last_clean_scrub_stamp":"2024-09-13T16:48:28.810988+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:12.536424+0000","scrub_duration":0.0035919440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"22'159","reported_seq":174,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:27.707816+0000","last_change":"2024-09-13T16:43:55.002041+0000","last_active":"2024-09-13T16:48:27.707816+0000","last_peered":"2024-09-13T16:48:27.707816+0000","last_clean":"2024-09-13T16:48:27.707816+0000","last_became_active":"2024-09-13T16:43:55.001512+0000","last_became_peered":"2024-09-13T16:43:55.001512+0000","last_unstale":"2024-09-13T16:48:27.707816+0000","last_undegraded":"2024-09-13T16:48:27.707816+0000","last_fullsized":"2024-09-13T16:48:27.707816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"22'4","reported_seq":31,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:31.538047+0000","last_change":"2024-09-13T16:48:31.538047+0000","last_active":"2024-09-13T16:48:31.538047+0000","last_peered":"2024-09-13T16:48:31.538047+0000","last_clean":"2024-09-13T16:48:31.538047+0000","last_became_active":"2024-09-13T16:43:54.550154+0000","last_became_peered":"2024-09-13T16:43:54.550154+0000","last_unstale":"2024-09-13T16:48:31.538047+0000","last_undegraded":"2024-09-13T16:48:31.538047+0000","last_fullsized":"2024-09-13T16:48:31.538047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'4","last_scrub_stamp":"2024-09-13T16:48:31.537892+0000","last_deep_scrub":"22'4","last_deep_scrub_stamp":"2024-09-13T16:48:31.537892+0000","last_clean_scrub_stamp":"2024-09-13T16:48:31.537892+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:09.435264+0000","scrub_duration":0.0031102679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"22'178","reported_seq":204,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:48.011383+0000","last_change":"2024-09-13T16:48:48.011383+0000","last_active":"2024-09-13T16:48:48.011383+0000","last_peered":"2024-09-13T16:48:48.011383+0000","last_clean":"2024-09-13T16:48:48.011383+0000","last_became_active":"2024-09-13T16:43:54.993036+0000","last_became_peered":"2024-09-13T16:43:54.993036+0000","last_unstale":"2024-09-13T16:48:48.011383+0000","last_undegraded":"2024-09-13T16:48:48.011383+0000","last_fullsized":"2024-09-13T16:48:48.011383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'178","last_scrub_stamp":"2024-09-13T16:48:48.011264+0000","last_deep_scrub":"22'178","last_deep_scrub_stamp":"2024-09-13T16:48:48.011264+0000","last_clean_scrub_stamp":"2024-09-13T16:48:48.011264+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:19.491965+0000","scrub_duration":0.001649017,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'8","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:40.755255+0000","last_change":"2024-09-13T16:48:40.755255+0000","last_active":"2024-09-13T16:48:40.755255+0000","last_peered":"2024-09-13T16:48:40.755255+0000","last_clean":"2024-09-13T16:48:40.755255+0000","last_became_active":"2024-09-13T16:43:54.555267+0000","last_became_peered":"2024-09-13T16:43:54.555267+0000","last_unstale":"2024-09-13T16:48:40.755255+0000","last_undegraded":"2024-09-13T16:48:40.755255+0000","last_fullsized":"2024-09-13T16:48:40.755255+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'8","last_scrub_stamp":"2024-09-13T16:48:40.755129+0000","last_deep_scrub":"22'8","last_deep_scrub_stamp":"2024-09-13T16:48:40.755129+0000","last_clean_scrub_stamp":"2024-09-13T16:48:40.755129+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:18.652500+0000","scrub_duration":0.0039776960000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"22'153","reported_seq":179,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:43.627266+0000","last_change":"2024-09-13T16:48:43.627266+0000","last_active":"2024-09-13T16:48:43.627266+0000","last_peered":"2024-09-13T16:48:43.627266+0000","last_clean":"2024-09-13T16:48:43.627266+0000","last_became_active":"2024-09-13T16:43:54.981429+0000","last_became_peered":"2024-09-13T16:43:54.981429+0000","last_unstale":"2024-09-13T16:48:43.627266+0000","last_undegraded":"2024-09-13T16:48:43.627266+0000","last_fullsized":"2024-09-13T16:48:43.627266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'153","last_scrub_stamp":"2024-09-13T16:48:43.627115+0000","last_deep_scrub":"22'153","last_deep_scrub_stamp":"2024-09-13T16:48:43.627115+0000","last_clean_scrub_stamp":"2024-09-13T16:48:43.627115+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:08:34.241232+0000","scrub_duration":0.001694222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'6","reported_seq":33,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:33.542809+0000","last_change":"2024-09-13T16:48:33.542809+0000","last_active":"2024-09-13T16:48:33.542809+0000","last_peered":"2024-09-13T16:48:33.542809+0000","last_clean":"2024-09-13T16:48:33.542809+0000","last_became_active":"2024-09-13T16:43:54.554307+0000","last_became_peered":"2024-09-13T16:43:54.554307+0000","last_unstale":"2024-09-13T16:48:33.542809+0000","last_undegraded":"2024-09-13T16:48:33.542809+0000","last_fullsized":"2024-09-13T16:48:33.542809+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-09-13T16:48:33.542675+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-09-13T16:48:33.542675+0000","last_clean_scrub_stamp":"2024-09-13T16:48:33.542675+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:22.957253+0000","scrub_duration":0.003084865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"22'204","reported_seq":219,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:25.991378+0000","last_change":"2024-09-13T16:43:54.993485+0000","last_active":"2024-09-13T16:48:25.991378+0000","last_peered":"2024-09-13T16:48:25.991378+0000","last_clean":"2024-09-13T16:48:25.991378+0000","last_became_active":"2024-09-13T16:43:54.993226+0000","last_became_peered":"2024-09-13T16:43:54.993226+0000","last_unstale":"2024-09-13T16:48:25.991378+0000","last_undegraded":"2024-09-13T16:48:25.991378+0000","last_fullsized":"2024-09-13T16:48:25.991378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'15","reported_seq":42,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:43.064614+0000","last_change":"2024-09-13T16:48:43.064614+0000","last_active":"2024-09-13T16:48:43.064614+0000","last_peered":"2024-09-13T16:48:43.064614+0000","last_clean":"2024-09-13T16:48:43.064614+0000","last_became_active":"2024-09-13T16:43:54.560480+0000","last_became_peered":"2024-09-13T16:43:54.560480+0000","last_unstale":"2024-09-13T16:48:43.064614+0000","last_undegraded":"2024-09-13T16:48:43.064614+0000","last_fullsized":"2024-09-13T16:48:43.064614+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-09-13T16:48:43.064502+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-09-13T16:48:43.064502+0000","last_clean_scrub_stamp":"2024-09-13T16:48:43.064502+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:22.155224+0000","scrub_duration":0.0045066539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"22'211","reported_seq":237,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:29.633706+0000","last_change":"2024-09-13T16:48:29.633706+0000","last_active":"2024-09-13T16:48:29.633706+0000","last_peered":"2024-09-13T16:48:29.633706+0000","last_clean":"2024-09-13T16:48:29.633706+0000","last_became_active":"2024-09-13T16:43:54.986083+0000","last_became_peered":"2024-09-13T16:43:54.986083+0000","last_unstale":"2024-09-13T16:48:29.633706+0000","last_undegraded":"2024-09-13T16:48:29.633706+0000","last_fullsized":"2024-09-13T16:48:29.633706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'211","last_scrub_stamp":"2024-09-13T16:48:29.633578+0000","last_deep_scrub":"22'211","last_deep_scrub_stamp":"2024-09-13T16:48:29.633578+0000","last_clean_scrub_stamp":"2024-09-13T16:48:29.633578+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:02.991594+0000","scrub_duration":0.001550811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'171","reported_seq":198,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:37.744830+0000","last_change":"2024-09-13T16:48:37.744926+0000","last_active":"2024-09-13T16:48:37.744830+0000","last_peered":"2024-09-13T16:48:37.744830+0000","last_clean":"2024-09-13T16:48:37.744830+0000","last_became_active":"2024-09-13T16:43:54.565701+0000","last_became_peered":"2024-09-13T16:43:54.565701+0000","last_unstale":"2024-09-13T16:48:37.744830+0000","last_undegraded":"2024-09-13T16:48:37.744830+0000","last_fullsized":"2024-09-13T16:48:37.744830+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'171","last_scrub_stamp":"2024-09-13T16:48:37.744806+0000","last_deep_scrub":"22'171","last_deep_scrub_stamp":"2024-09-13T16:48:37.744806+0000","last_clean_scrub_stamp":"2024-09-13T16:48:37.744806+0000","objects_scrubbed":6,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:10:01.111758+0000","scrub_duration":0.0062340260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364698,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"22'241","reported_seq":256,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:29.997907+0000","last_change":"2024-09-13T16:43:54.989068+0000","last_active":"2024-09-13T16:48:29.997907+0000","last_peered":"2024-09-13T16:48:29.997907+0000","last_clean":"2024-09-13T16:48:29.997907+0000","last_became_active":"2024-09-13T16:43:54.988779+0000","last_became_peered":"2024-09-13T16:43:54.988779+0000","last_unstale":"2024-09-13T16:48:29.997907+0000","last_undegraded":"2024-09-13T16:48:29.997907+0000","last_fullsized":"2024-09-13T16:48:29.997907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":213,"num_write_kb":581632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'15","reported_seq":42,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:28.421871+0000","last_change":"2024-09-13T16:48:28.421871+0000","last_active":"2024-09-13T16:48:28.421871+0000","last_peered":"2024-09-13T16:48:28.421871+0000","last_clean":"2024-09-13T16:48:28.421871+0000","last_became_active":"2024-09-13T16:43:54.561077+0000","last_became_peered":"2024-09-13T16:43:54.561077+0000","last_unstale":"2024-09-13T16:48:28.421871+0000","last_undegraded":"2024-09-13T16:48:28.421871+0000","last_fullsized":"2024-09-13T16:48:28.421871+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-09-13T16:48:28.421778+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-09-13T16:48:28.421778+0000","last_clean_scrub_stamp":"2024-09-13T16:48:28.421778+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:09.651705+0000","scrub_duration":0.0051464910000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"22'162","reported_seq":188,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:31.409973+0000","last_change":"2024-09-13T16:48:31.409973+0000","last_active":"2024-09-13T16:48:31.409973+0000","last_peered":"2024-09-13T16:48:31.409973+0000","last_clean":"2024-09-13T16:48:31.409973+0000","last_became_active":"2024-09-13T16:43:54.999283+0000","last_became_peered":"2024-09-13T16:43:54.999283+0000","last_unstale":"2024-09-13T16:48:31.409973+0000","last_undegraded":"2024-09-13T16:48:31.409973+0000","last_fullsized":"2024-09-13T16:48:31.409973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'162","last_scrub_stamp":"2024-09-13T16:48:31.409830+0000","last_deep_scrub":"22'162","last_deep_scrub_stamp":"2024-09-13T16:48:31.409830+0000","last_clean_scrub_stamp":"2024-09-13T16:48:31.409830+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:15.135267+0000","scrub_duration":0.002545552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"22'8","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:38.572732+0000","last_change":"2024-09-13T16:48:38.572847+0000","last_active":"2024-09-13T16:48:38.572732+0000","last_peered":"2024-09-13T16:48:38.572732+0000","last_clean":"2024-09-13T16:48:38.572732+0000","last_became_active":"2024-09-13T16:43:54.542548+0000","last_became_peered":"2024-09-13T16:43:54.542548+0000","last_unstale":"2024-09-13T16:48:38.572732+0000","last_undegraded":"2024-09-13T16:48:38.572732+0000","last_fullsized":"2024-09-13T16:48:38.572732+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'8","last_scrub_stamp":"2024-09-13T16:48:38.572702+0000","last_deep_scrub":"22'8","last_deep_scrub_stamp":"2024-09-13T16:48:38.572702+0000","last_clean_scrub_stamp":"2024-09-13T16:48:38.572702+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:36.584734+0000","scrub_duration":0.0044139139999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"22'195","reported_seq":210,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:29.997883+0000","last_change":"2024-09-13T16:43:54.989168+0000","last_active":"2024-09-13T16:48:29.997883+0000","last_peered":"2024-09-13T16:48:29.997883+0000","last_clean":"2024-09-13T16:48:29.997883+0000","last_became_active":"2024-09-13T16:43:54.988716+0000","last_became_peered":"2024-09-13T16:43:54.988716+0000","last_unstale":"2024-09-13T16:48:29.997883+0000","last_undegraded":"2024-09-13T16:48:29.997883+0000","last_fullsized":"2024-09-13T16:48:29.997883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'9","reported_seq":36,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:35.743470+0000","last_change":"2024-09-13T16:48:35.743577+0000","last_active":"2024-09-13T16:48:35.743470+0000","last_peered":"2024-09-13T16:48:35.743470+0000","last_clean":"2024-09-13T16:48:35.743470+0000","last_became_active":"2024-09-13T16:43:54.551293+0000","last_became_peered":"2024-09-13T16:43:54.551293+0000","last_unstale":"2024-09-13T16:48:35.743470+0000","last_undegraded":"2024-09-13T16:48:35.743470+0000","last_fullsized":"2024-09-13T16:48:35.743470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-09-13T16:48:35.743447+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-09-13T16:48:35.743447+0000","last_clean_scrub_stamp":"2024-09-13T16:48:35.743447+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:19.468883+0000","scrub_duration":0.0033744299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":940,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"22'169","reported_seq":195,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:41.738426+0000","last_change":"2024-09-13T16:48:41.738532+0000","last_active":"2024-09-13T16:48:41.738426+0000","last_peered":"2024-09-13T16:48:41.738426+0000","last_clean":"2024-09-13T16:48:41.738426+0000","last_became_active":"2024-09-13T16:43:54.994245+0000","last_became_peered":"2024-09-13T16:43:54.994245+0000","last_unstale":"2024-09-13T16:48:41.738426+0000","last_undegraded":"2024-09-13T16:48:41.738426+0000","last_fullsized":"2024-09-13T16:48:41.738426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'169","last_scrub_stamp":"2024-09-13T16:48:41.738403+0000","last_deep_scrub":"22'169","last_deep_scrub_stamp":"2024-09-13T16:48:41.738403+0000","last_clean_scrub_stamp":"2024-09-13T16:48:41.738403+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:30.637627+0000","scrub_duration":0.002340941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'26","reported_seq":53,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:34.429476+0000","last_change":"2024-09-13T16:48:34.429476+0000","last_active":"2024-09-13T16:48:34.429476+0000","last_peered":"2024-09-13T16:48:34.429476+0000","last_clean":"2024-09-13T16:48:34.429476+0000","last_became_active":"2024-09-13T16:43:54.561939+0000","last_became_peered":"2024-09-13T16:43:54.561939+0000","last_unstale":"2024-09-13T16:48:34.429476+0000","last_undegraded":"2024-09-13T16:48:34.429476+0000","last_fullsized":"2024-09-13T16:48:34.429476+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'26","last_scrub_stamp":"2024-09-13T16:48:34.429340+0000","last_deep_scrub":"22'26","last_deep_scrub_stamp":"2024-09-13T16:48:34.429340+0000","last_clean_scrub_stamp":"2024-09-13T16:48:34.429340+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:18.154777+0000","scrub_duration":0.006433701,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"22'175","reported_seq":201,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:48.910813+0000","last_change":"2024-09-13T16:48:48.910907+0000","last_active":"2024-09-13T16:48:48.910813+0000","last_peered":"2024-09-13T16:48:48.910813+0000","last_clean":"2024-09-13T16:48:48.910813+0000","last_became_active":"2024-09-13T16:43:54.988172+0000","last_became_peered":"2024-09-13T16:43:54.988172+0000","last_unstale":"2024-09-13T16:48:48.910813+0000","last_undegraded":"2024-09-13T16:48:48.910813+0000","last_fullsized":"2024-09-13T16:48:48.910813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'175","last_scrub_stamp":"2024-09-13T16:48:48.910789+0000","last_deep_scrub":"22'175","last_deep_scrub_stamp":"2024-09-13T16:48:48.910789+0000","last_clean_scrub_stamp":"2024-09-13T16:48:48.910789+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:16.660275+0000","scrub_duration":0.0015205030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'5","reported_seq":32,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:28.623118+0000","last_change":"2024-09-13T16:48:28.623229+0000","last_active":"2024-09-13T16:48:28.623118+0000","last_peered":"2024-09-13T16:48:28.623118+0000","last_clean":"2024-09-13T16:48:28.623118+0000","last_became_active":"2024-09-13T16:43:54.559111+0000","last_became_peered":"2024-09-13T16:43:54.559111+0000","last_unstale":"2024-09-13T16:48:28.623118+0000","last_undegraded":"2024-09-13T16:48:28.623118+0000","last_fullsized":"2024-09-13T16:48:28.623118+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'5","last_scrub_stamp":"2024-09-13T16:48:28.623093+0000","last_deep_scrub":"22'5","last_deep_scrub_stamp":"2024-09-13T16:48:28.623093+0000","last_clean_scrub_stamp":"2024-09-13T16:48:28.623093+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:43.512775+0000","scrub_duration":0.0030479550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'156","reported_seq":171,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:29.528109+0000","last_change":"2024-09-13T16:43:54.988313+0000","last_active":"2024-09-13T16:48:29.528109+0000","last_peered":"2024-09-13T16:48:29.528109+0000","last_clean":"2024-09-13T16:48:29.528109+0000","last_became_active":"2024-09-13T16:43:54.988059+0000","last_became_peered":"2024-09-13T16:43:54.988059+0000","last_unstale":"2024-09-13T16:48:29.528109+0000","last_undegraded":"2024-09-13T16:48:29.528109+0000","last_fullsized":"2024-09-13T16:48:29.528109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'7","reported_seq":34,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:39.488130+0000","last_change":"2024-09-13T16:48:39.488234+0000","last_active":"2024-09-13T16:48:39.488130+0000","last_peered":"2024-09-13T16:48:39.488130+0000","last_clean":"2024-09-13T16:48:39.488130+0000","last_became_active":"2024-09-13T16:43:54.565861+0000","last_became_peered":"2024-09-13T16:43:54.565861+0000","last_unstale":"2024-09-13T16:48:39.488130+0000","last_undegraded":"2024-09-13T16:48:39.488130+0000","last_fullsized":"2024-09-13T16:48:39.488130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'7","last_scrub_stamp":"2024-09-13T16:48:39.488109+0000","last_deep_scrub":"22'7","last_deep_scrub_stamp":"2024-09-13T16:48:39.488109+0000","last_clean_scrub_stamp":"2024-09-13T16:48:39.488109+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:17.385480+0000","scrub_duration":0.0035285960000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"22'136","reported_seq":148,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.339101+0000","last_change":"2024-09-13T16:44:58.956863+0000","last_active":"2024-09-13T16:48:22.339101+0000","last_peered":"2024-09-13T16:48:22.339101+0000","last_clean":"2024-09-13T16:48:22.339101+0000","last_became_active":"2024-09-13T16:44:58.956557+0000","last_became_peered":"2024-09-13T16:44:58.956557+0000","last_unstale":"2024-09-13T16:48:22.339101+0000","last_undegraded":"2024-09-13T16:48:22.339101+0000","last_fullsized":"2024-09-13T16:48:22.339101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":278528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"22'4","reported_seq":31,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:36.093320+0000","last_change":"2024-09-13T16:48:36.093416+0000","last_active":"2024-09-13T16:48:36.093320+0000","last_peered":"2024-09-13T16:48:36.093320+0000","last_clean":"2024-09-13T16:48:36.093320+0000","last_became_active":"2024-09-13T16:43:54.559353+0000","last_became_peered":"2024-09-13T16:43:54.559353+0000","last_unstale":"2024-09-13T16:48:36.093320+0000","last_undegraded":"2024-09-13T16:48:36.093320+0000","last_fullsized":"2024-09-13T16:48:36.093320+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'4","last_scrub_stamp":"2024-09-13T16:48:36.093289+0000","last_deep_scrub":"22'4","last_deep_scrub_stamp":"2024-09-13T16:48:36.093289+0000","last_clean_scrub_stamp":"2024-09-13T16:48:36.093289+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:27.023954+0000","scrub_duration":0.0036442430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"22'209","reported_seq":224,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:25.519073+0000","last_change":"2024-09-13T16:43:54.992511+0000","last_active":"2024-09-13T16:48:25.519073+0000","last_peered":"2024-09-13T16:48:25.519073+0000","last_clean":"2024-09-13T16:48:25.519073+0000","last_became_active":"2024-09-13T16:43:54.992249+0000","last_became_peered":"2024-09-13T16:43:54.992249+0000","last_unstale":"2024-09-13T16:48:25.519073+0000","last_undegraded":"2024-09-13T16:48:25.519073+0000","last_fullsized":"2024-09-13T16:48:25.519073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'8","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:30.390613+0000","last_change":"2024-09-13T16:48:30.390713+0000","last_active":"2024-09-13T16:48:30.390613+0000","last_peered":"2024-09-13T16:48:30.390613+0000","last_clean":"2024-09-13T16:48:30.390613+0000","last_became_active":"2024-09-13T16:43:54.547074+0000","last_became_peered":"2024-09-13T16:43:54.547074+0000","last_unstale":"2024-09-13T16:48:30.390613+0000","last_undegraded":"2024-09-13T16:48:30.390613+0000","last_fullsized":"2024-09-13T16:48:30.390613+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'8","last_scrub_stamp":"2024-09-13T16:48:30.390596+0000","last_deep_scrub":"22'8","last_deep_scrub_stamp":"2024-09-13T16:48:30.390596+0000","last_clean_scrub_stamp":"2024-09-13T16:48:30.390596+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:11.620523+0000","scrub_duration":0.0040624839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"22'172","reported_seq":187,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:29.528066+0000","last_change":"2024-09-13T16:43:54.990458+0000","last_active":"2024-09-13T16:48:29.528066+0000","last_peered":"2024-09-13T16:48:29.528066+0000","last_clean":"2024-09-13T16:48:29.528066+0000","last_became_active":"2024-09-13T16:43:54.990196+0000","last_became_peered":"2024-09-13T16:43:54.990196+0000","last_unstale":"2024-09-13T16:48:29.528066+0000","last_undegraded":"2024-09-13T16:48:29.528066+0000","last_fullsized":"2024-09-13T16:48:29.528066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'119","reported_seq":182,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:35.961536+0000","last_change":"2024-09-13T16:48:35.961673+0000","last_active":"2024-09-13T16:48:35.961536+0000","last_peered":"2024-09-13T16:48:35.961536+0000","last_clean":"2024-09-13T16:48:35.961536+0000","last_became_active":"2024-09-13T16:43:51.913797+0000","last_became_peered":"2024-09-13T16:43:51.913797+0000","last_unstale":"2024-09-13T16:48:35.961536+0000","last_undegraded":"2024-09-13T16:48:35.961536+0000","last_fullsized":"2024-09-13T16:48:35.961536+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'119","last_scrub_stamp":"2024-09-13T16:48:35.961513+0000","last_deep_scrub":"22'119","last_deep_scrub_stamp":"2024-09-13T16:48:35.961513+0000","last_clean_scrub_stamp":"2024-09-13T16:48:35.961513+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:59:32.930667+0000","scrub_duration":0.0053261610000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"22'11","reported_seq":38,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:40.552690+0000","last_change":"2024-09-13T16:48:40.552797+0000","last_active":"2024-09-13T16:48:40.552690+0000","last_peered":"2024-09-13T16:48:40.552690+0000","last_clean":"2024-09-13T16:48:40.552690+0000","last_became_active":"2024-09-13T16:43:54.563949+0000","last_became_peered":"2024-09-13T16:43:54.563949+0000","last_unstale":"2024-09-13T16:48:40.552690+0000","last_undegraded":"2024-09-13T16:48:40.552690+0000","last_fullsized":"2024-09-13T16:48:40.552690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'11","last_scrub_stamp":"2024-09-13T16:48:40.552667+0000","last_deep_scrub":"22'11","last_deep_scrub_stamp":"2024-09-13T16:48:40.552667+0000","last_clean_scrub_stamp":"2024-09-13T16:48:40.552667+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:19.643388+0000","scrub_duration":0.004950004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"22'215","reported_seq":238,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:48.528055+0000","last_change":"2024-09-13T16:48:48.528169+0000","last_active":"2024-09-13T16:48:48.528055+0000","last_peered":"2024-09-13T16:48:48.528055+0000","last_clean":"2024-09-13T16:48:48.528055+0000","last_became_active":"2024-09-13T16:44:59.386756+0000","last_became_peered":"2024-09-13T16:44:59.386756+0000","last_unstale":"2024-09-13T16:48:48.528055+0000","last_undegraded":"2024-09-13T16:48:48.528055+0000","last_fullsized":"2024-09-13T16:48:48.528055+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'215","last_scrub_stamp":"2024-09-13T16:48:48.528033+0000","last_deep_scrub":"22'215","last_deep_scrub_stamp":"2024-09-13T16:48:48.528033+0000","last_clean_scrub_stamp":"2024-09-13T16:48:48.528033+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:27.618755+0000","scrub_duration":0.0021049049999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":186,"num_write_kb":507904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"22'5","reported_seq":32,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:37.992778+0000","last_change":"2024-09-13T16:48:37.992883+0000","last_active":"2024-09-13T16:48:37.992778+0000","last_peered":"2024-09-13T16:48:37.992778+0000","last_clean":"2024-09-13T16:48:37.992778+0000","last_became_active":"2024-09-13T16:43:54.564605+0000","last_became_peered":"2024-09-13T16:43:54.564605+0000","last_unstale":"2024-09-13T16:48:37.992778+0000","last_undegraded":"2024-09-13T16:48:37.992778+0000","last_fullsized":"2024-09-13T16:48:37.992778+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'5","last_scrub_stamp":"2024-09-13T16:48:37.992756+0000","last_deep_scrub":"22'5","last_deep_scrub_stamp":"2024-09-13T16:48:37.992756+0000","last_clean_scrub_stamp":"2024-09-13T16:48:37.992756+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:28.923420+0000","scrub_duration":0.0036587429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"22'192","reported_seq":218,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:33.451090+0000","last_change":"2024-09-13T16:48:33.451217+0000","last_active":"2024-09-13T16:48:33.451090+0000","last_peered":"2024-09-13T16:48:33.451090+0000","last_clean":"2024-09-13T16:48:33.451090+0000","last_became_active":"2024-09-13T16:43:54.997510+0000","last_became_peered":"2024-09-13T16:43:54.997510+0000","last_unstale":"2024-09-13T16:48:33.451090+0000","last_undegraded":"2024-09-13T16:48:33.451090+0000","last_fullsized":"2024-09-13T16:48:33.451090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'192","last_scrub_stamp":"2024-09-13T16:48:33.451068+0000","last_deep_scrub":"22'192","last_deep_scrub_stamp":"2024-09-13T16:48:33.451068+0000","last_clean_scrub_stamp":"2024-09-13T16:48:33.451068+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:09:56.818020+0000","scrub_duration":0.0021645589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'15","reported_seq":47,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:36.404304+0000","last_change":"2024-09-13T16:48:36.404425+0000","last_active":"2024-09-13T16:48:36.404304+0000","last_peered":"2024-09-13T16:48:36.404304+0000","last_clean":"2024-09-13T16:48:36.404304+0000","last_became_active":"2024-09-13T16:43:54.566023+0000","last_became_peered":"2024-09-13T16:43:54.566023+0000","last_unstale":"2024-09-13T16:48:36.404304+0000","last_undegraded":"2024-09-13T16:48:36.404304+0000","last_fullsized":"2024-09-13T16:48:36.404304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-09-13T16:48:36.404278+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-09-13T16:48:36.404278+0000","last_clean_scrub_stamp":"2024-09-13T16:48:36.404278+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:09:59.771230+0000","scrub_duration":0.0040041060000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"22'167","reported_seq":193,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:45.035817+0000","last_change":"2024-09-13T16:48:45.035919+0000","last_active":"2024-09-13T16:48:45.035817+0000","last_peered":"2024-09-13T16:48:45.035817+0000","last_clean":"2024-09-13T16:48:45.035817+0000","last_became_active":"2024-09-13T16:43:54.980766+0000","last_became_peered":"2024-09-13T16:43:54.980766+0000","last_unstale":"2024-09-13T16:48:45.035817+0000","last_undegraded":"2024-09-13T16:48:45.035817+0000","last_fullsized":"2024-09-13T16:48:45.035817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'167","last_scrub_stamp":"2024-09-13T16:48:45.035795+0000","last_deep_scrub":"22'167","last_deep_scrub_stamp":"2024-09-13T16:48:45.035795+0000","last_clean_scrub_stamp":"2024-09-13T16:48:45.035795+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:09.803222+0000","scrub_duration":0.001815189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"22'7","reported_seq":34,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:32.563054+0000","last_change":"2024-09-13T16:48:32.563164+0000","last_active":"2024-09-13T16:48:32.563054+0000","last_peered":"2024-09-13T16:48:32.563054+0000","last_clean":"2024-09-13T16:48:32.563054+0000","last_became_active":"2024-09-13T16:43:54.563522+0000","last_became_peered":"2024-09-13T16:43:54.563522+0000","last_unstale":"2024-09-13T16:48:32.563054+0000","last_undegraded":"2024-09-13T16:48:32.563054+0000","last_fullsized":"2024-09-13T16:48:32.563054+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'7","last_scrub_stamp":"2024-09-13T16:48:32.563029+0000","last_deep_scrub":"22'7","last_deep_scrub_stamp":"2024-09-13T16:48:32.563029+0000","last_clean_scrub_stamp":"2024-09-13T16:48:32.563029+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:21.462253+0000","scrub_duration":0.0038373040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"22'163","reported_seq":189,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:30.616538+0000","last_change":"2024-09-13T16:48:30.616641+0000","last_active":"2024-09-13T16:48:30.616538+0000","last_peered":"2024-09-13T16:48:30.616538+0000","last_clean":"2024-09-13T16:48:30.616538+0000","last_became_active":"2024-09-13T16:43:54.998803+0000","last_became_peered":"2024-09-13T16:43:54.998803+0000","last_unstale":"2024-09-13T16:48:30.616538+0000","last_undegraded":"2024-09-13T16:48:30.616538+0000","last_fullsized":"2024-09-13T16:48:30.616538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'163","last_scrub_stamp":"2024-09-13T16:48:30.616512+0000","last_deep_scrub":"22'163","last_deep_scrub_stamp":"2024-09-13T16:48:30.616512+0000","last_clean_scrub_stamp":"2024-09-13T16:48:30.616512+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:33:30.274099+0000","scrub_duration":0.0019863279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":28,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:40.074852+0000","last_change":"2024-09-13T16:48:40.075007+0000","last_active":"2024-09-13T16:48:40.074852+0000","last_peered":"2024-09-13T16:48:40.074852+0000","last_clean":"2024-09-13T16:48:40.074852+0000","last_became_active":"2024-09-13T16:43:54.568846+0000","last_became_peered":"2024-09-13T16:43:54.568846+0000","last_unstale":"2024-09-13T16:48:40.074852+0000","last_undegraded":"2024-09-13T16:48:40.074852+0000","last_fullsized":"2024-09-13T16:48:40.074852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'1","last_scrub_stamp":"2024-09-13T16:48:40.074829+0000","last_deep_scrub":"20'1","last_deep_scrub_stamp":"2024-09-13T16:48:40.074829+0000","last_clean_scrub_stamp":"2024-09-13T16:48:40.074829+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:36:47.384962+0000","scrub_duration":0.001876604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"22'179","reported_seq":194,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.349894+0000","last_change":"2024-09-13T16:43:54.992082+0000","last_active":"2024-09-13T16:48:22.349894+0000","last_peered":"2024-09-13T16:48:22.349894+0000","last_clean":"2024-09-13T16:48:22.349894+0000","last_became_active":"2024-09-13T16:43:54.991659+0000","last_became_peered":"2024-09-13T16:43:54.991659+0000","last_unstale":"2024-09-13T16:48:22.349894+0000","last_undegraded":"2024-09-13T16:48:22.349894+0000","last_fullsized":"2024-09-13T16:48:22.349894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"22'7","reported_seq":34,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:44.922669+0000","last_change":"2024-09-13T16:48:44.922788+0000","last_active":"2024-09-13T16:48:44.922669+0000","last_peered":"2024-09-13T16:48:44.922669+0000","last_clean":"2024-09-13T16:48:44.922669+0000","last_became_active":"2024-09-13T16:43:54.568903+0000","last_became_peered":"2024-09-13T16:43:54.568903+0000","last_unstale":"2024-09-13T16:48:44.922669+0000","last_undegraded":"2024-09-13T16:48:44.922669+0000","last_fullsized":"2024-09-13T16:48:44.922669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'7","last_scrub_stamp":"2024-09-13T16:48:44.922644+0000","last_deep_scrub":"22'7","last_deep_scrub_stamp":"2024-09-13T16:48:44.922644+0000","last_clean_scrub_stamp":"2024-09-13T16:48:44.922644+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:24.013365+0000","scrub_duration":0.0046489829999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"22'206","reported_seq":232,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:47.016450+0000","last_change":"2024-09-13T16:48:47.016564+0000","last_active":"2024-09-13T16:48:47.016450+0000","last_peered":"2024-09-13T16:48:47.016450+0000","last_clean":"2024-09-13T16:48:47.016450+0000","last_became_active":"2024-09-13T16:43:54.979584+0000","last_became_peered":"2024-09-13T16:43:54.979584+0000","last_unstale":"2024-09-13T16:48:47.016450+0000","last_undegraded":"2024-09-13T16:48:47.016450+0000","last_fullsized":"2024-09-13T16:48:47.016450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'206","last_scrub_stamp":"2024-09-13T16:48:47.016426+0000","last_deep_scrub":"22'206","last_deep_scrub_stamp":"2024-09-13T16:48:47.016426+0000","last_clean_scrub_stamp":"2024-09-13T16:48:47.016426+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:14.765912+0000","scrub_duration":0.001764435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'24","reported_seq":51,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:34.077498+0000","last_change":"2024-09-13T16:48:34.077608+0000","last_active":"2024-09-13T16:48:34.077498+0000","last_peered":"2024-09-13T16:48:34.077498+0000","last_clean":"2024-09-13T16:48:34.077498+0000","last_became_active":"2024-09-13T16:43:54.560030+0000","last_became_peered":"2024-09-13T16:43:54.560030+0000","last_unstale":"2024-09-13T16:48:34.077498+0000","last_undegraded":"2024-09-13T16:48:34.077498+0000","last_fullsized":"2024-09-13T16:48:34.077498+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-09-13T16:48:34.077471+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-09-13T16:48:34.077471+0000","last_clean_scrub_stamp":"2024-09-13T16:48:34.077471+0000","objects_scrubbed":4,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:59:31.046625+0000","scrub_duration":0.003834367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'196","reported_seq":222,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:43.939466+0000","last_change":"2024-09-13T16:48:43.939576+0000","last_active":"2024-09-13T16:48:43.939466+0000","last_peered":"2024-09-13T16:48:43.939466+0000","last_clean":"2024-09-13T16:48:43.939466+0000","last_became_active":"2024-09-13T16:43:54.990440+0000","last_became_peered":"2024-09-13T16:43:54.990440+0000","last_unstale":"2024-09-13T16:48:43.939466+0000","last_undegraded":"2024-09-13T16:48:43.939466+0000","last_fullsized":"2024-09-13T16:48:43.939466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'196","last_scrub_stamp":"2024-09-13T16:48:43.939446+0000","last_deep_scrub":"22'196","last_deep_scrub_stamp":"2024-09-13T16:48:43.939446+0000","last_clean_scrub_stamp":"2024-09-13T16:48:43.939446+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:23.030168+0000","scrub_duration":0.002075665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'12","reported_seq":39,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:29.809769+0000","last_change":"2024-09-13T16:48:29.809876+0000","last_active":"2024-09-13T16:48:29.809769+0000","last_peered":"2024-09-13T16:48:29.809769+0000","last_clean":"2024-09-13T16:48:29.809769+0000","last_became_active":"2024-09-13T16:43:54.558194+0000","last_became_peered":"2024-09-13T16:43:54.558194+0000","last_unstale":"2024-09-13T16:48:29.809769+0000","last_undegraded":"2024-09-13T16:48:29.809769+0000","last_fullsized":"2024-09-13T16:48:29.809769+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-09-13T16:48:29.809742+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-09-13T16:48:29.809742+0000","last_clean_scrub_stamp":"2024-09-13T16:48:29.809742+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:33:29.467329+0000","scrub_duration":0.0040895360000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"22'161","reported_seq":187,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:37.486307+0000","last_change":"2024-09-13T16:48:37.486418+0000","last_active":"2024-09-13T16:48:37.486307+0000","last_peered":"2024-09-13T16:48:37.486307+0000","last_clean":"2024-09-13T16:48:37.486307+0000","last_became_active":"2024-09-13T16:43:54.995397+0000","last_became_peered":"2024-09-13T16:43:54.995397+0000","last_unstale":"2024-09-13T16:48:37.486307+0000","last_undegraded":"2024-09-13T16:48:37.486307+0000","last_fullsized":"2024-09-13T16:48:37.486307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'161","last_scrub_stamp":"2024-09-13T16:48:37.486283+0000","last_deep_scrub":"22'161","last_deep_scrub_stamp":"2024-09-13T16:48:37.486283+0000","last_clean_scrub_stamp":"2024-09-13T16:48:37.486283+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:26.385507+0000","scrub_duration":0.0018563010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"22'3","reported_seq":30,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:47.903413+0000","last_change":"2024-09-13T16:48:47.903511+0000","last_active":"2024-09-13T16:48:47.903413+0000","last_peered":"2024-09-13T16:48:47.903413+0000","last_clean":"2024-09-13T16:48:47.903413+0000","last_became_active":"2024-09-13T16:43:54.568892+0000","last_became_peered":"2024-09-13T16:43:54.568892+0000","last_unstale":"2024-09-13T16:48:47.903413+0000","last_undegraded":"2024-09-13T16:48:47.903413+0000","last_fullsized":"2024-09-13T16:48:47.903413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-09-13T16:48:47.903389+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-09-13T16:48:47.903389+0000","last_clean_scrub_stamp":"2024-09-13T16:48:47.903389+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:02:49.388871+0000","scrub_duration":0.0037439740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"22'198","reported_seq":224,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:31.599065+0000","last_change":"2024-09-13T16:48:31.599179+0000","last_active":"2024-09-13T16:48:31.599065+0000","last_peered":"2024-09-13T16:48:31.599065+0000","last_clean":"2024-09-13T16:48:31.599065+0000","last_became_active":"2024-09-13T16:43:54.983886+0000","last_became_peered":"2024-09-13T16:43:54.983886+0000","last_unstale":"2024-09-13T16:48:31.599065+0000","last_undegraded":"2024-09-13T16:48:31.599065+0000","last_fullsized":"2024-09-13T16:48:31.599065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'198","last_scrub_stamp":"2024-09-13T16:48:31.599041+0000","last_deep_scrub":"22'198","last_deep_scrub_stamp":"2024-09-13T16:48:31.599041+0000","last_clean_scrub_stamp":"2024-09-13T16:48:31.599041+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:55:15.644305+0000","scrub_duration":0.0022504479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"22'11","reported_seq":38,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:32.092037+0000","last_change":"2024-09-13T16:48:32.092150+0000","last_active":"2024-09-13T16:48:32.092037+0000","last_peered":"2024-09-13T16:48:32.092037+0000","last_clean":"2024-09-13T16:48:32.092037+0000","last_became_active":"2024-09-13T16:43:54.554230+0000","last_became_peered":"2024-09-13T16:43:54.554230+0000","last_unstale":"2024-09-13T16:48:32.092037+0000","last_undegraded":"2024-09-13T16:48:32.092037+0000","last_fullsized":"2024-09-13T16:48:32.092037+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'11","last_scrub_stamp":"2024-09-13T16:48:32.092012+0000","last_deep_scrub":"22'11","last_deep_scrub_stamp":"2024-09-13T16:48:32.092012+0000","last_clean_scrub_stamp":"2024-09-13T16:48:32.092012+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:08.810087+0000","scrub_duration":0.0049653240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"22'178","reported_seq":204,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:48.902978+0000","last_change":"2024-09-13T16:48:48.903084+0000","last_active":"2024-09-13T16:48:48.902978+0000","last_peered":"2024-09-13T16:48:48.902978+0000","last_clean":"2024-09-13T16:48:48.902978+0000","last_became_active":"2024-09-13T16:43:54.984221+0000","last_became_peered":"2024-09-13T16:43:54.984221+0000","last_unstale":"2024-09-13T16:48:48.902978+0000","last_undegraded":"2024-09-13T16:48:48.902978+0000","last_fullsized":"2024-09-13T16:48:48.902978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'178","last_scrub_stamp":"2024-09-13T16:48:48.902943+0000","last_deep_scrub":"22'178","last_deep_scrub_stamp":"2024-09-13T16:48:48.902943+0000","last_clean_scrub_stamp":"2024-09-13T16:48:48.902943+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:06.299355+0000","scrub_duration":0.001721457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"22'5","reported_seq":32,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:35.116763+0000","last_change":"2024-09-13T16:48:35.116864+0000","last_active":"2024-09-13T16:48:35.116763+0000","last_peered":"2024-09-13T16:48:35.116763+0000","last_clean":"2024-09-13T16:48:35.116763+0000","last_became_active":"2024-09-13T16:43:54.556888+0000","last_became_peered":"2024-09-13T16:43:54.556888+0000","last_unstale":"2024-09-13T16:48:35.116763+0000","last_undegraded":"2024-09-13T16:48:35.116763+0000","last_fullsized":"2024-09-13T16:48:35.116763+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'5","last_scrub_stamp":"2024-09-13T16:48:35.116742+0000","last_deep_scrub":"22'5","last_deep_scrub_stamp":"2024-09-13T16:48:35.116742+0000","last_clean_scrub_stamp":"2024-09-13T16:48:35.116742+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:05.443913+0000","scrub_duration":0.0036882849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"22'217","reported_seq":243,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:35.884068+0000","last_change":"2024-09-13T16:48:35.884174+0000","last_active":"2024-09-13T16:48:35.884068+0000","last_peered":"2024-09-13T16:48:35.884068+0000","last_clean":"2024-09-13T16:48:35.884068+0000","last_became_active":"2024-09-13T16:43:54.999101+0000","last_became_peered":"2024-09-13T16:43:54.999101+0000","last_unstale":"2024-09-13T16:48:35.884068+0000","last_undegraded":"2024-09-13T16:48:35.884068+0000","last_fullsized":"2024-09-13T16:48:35.884068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'217","last_scrub_stamp":"2024-09-13T16:48:35.884045+0000","last_deep_scrub":"22'217","last_deep_scrub_stamp":"2024-09-13T16:48:35.884045+0000","last_clean_scrub_stamp":"2024-09-13T16:48:35.884045+0000","objects_scrubbed":0,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:24.783269+0000","scrub_duration":0.0018987959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'7","reported_seq":34,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:41.928025+0000","last_change":"2024-09-13T16:48:41.928129+0000","last_active":"2024-09-13T16:48:41.928025+0000","last_peered":"2024-09-13T16:48:41.928025+0000","last_clean":"2024-09-13T16:48:41.928025+0000","last_became_active":"2024-09-13T16:43:54.558233+0000","last_became_peered":"2024-09-13T16:43:54.558233+0000","last_unstale":"2024-09-13T16:48:41.928025+0000","last_undegraded":"2024-09-13T16:48:41.928025+0000","last_fullsized":"2024-09-13T16:48:41.928025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'7","last_scrub_stamp":"2024-09-13T16:48:41.927998+0000","last_deep_scrub":"22'7","last_deep_scrub_stamp":"2024-09-13T16:48:41.927998+0000","last_clean_scrub_stamp":"2024-09-13T16:48:41.927998+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:36:49.238131+0000","scrub_duration":0.0029359780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"22'190","reported_seq":205,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.327923+0000","last_change":"2024-09-13T16:43:54.986565+0000","last_active":"2024-09-13T16:48:22.327923+0000","last_peered":"2024-09-13T16:48:22.327923+0000","last_clean":"2024-09-13T16:48:22.327923+0000","last_became_active":"2024-09-13T16:43:54.986084+0000","last_became_peered":"2024-09-13T16:43:54.986084+0000","last_unstale":"2024-09-13T16:48:22.327923+0000","last_undegraded":"2024-09-13T16:48:22.327923+0000","last_fullsized":"2024-09-13T16:48:22.327923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"22'165","reported_seq":179,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.349998+0000","last_change":"2024-09-13T16:43:54.982611+0000","last_active":"2024-09-13T16:48:22.349998+0000","last_peered":"2024-09-13T16:48:22.349998+0000","last_clean":"2024-09-13T16:48:22.349998+0000","last_became_active":"2024-09-13T16:43:54.982338+0000","last_became_peered":"2024-09-13T16:43:54.982338+0000","last_unstale":"2024-09-13T16:48:22.349998+0000","last_undegraded":"2024-09-13T16:48:22.349998+0000","last_fullsized":"2024-09-13T16:48:22.349998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'8","reported_seq":34,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:36.989375+0000","last_change":"2024-09-13T16:48:36.989486+0000","last_active":"2024-09-13T16:48:36.989375+0000","last_peered":"2024-09-13T16:48:36.989375+0000","last_clean":"2024-09-13T16:48:36.989375+0000","last_became_active":"2024-09-13T16:43:54.558332+0000","last_became_peered":"2024-09-13T16:43:54.558332+0000","last_unstale":"2024-09-13T16:48:36.989375+0000","last_undegraded":"2024-09-13T16:48:36.989375+0000","last_fullsized":"2024-09-13T16:48:36.989375+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'8","last_scrub_stamp":"2024-09-13T16:48:36.989352+0000","last_deep_scrub":"22'8","last_deep_scrub_stamp":"2024-09-13T16:48:36.989352+0000","last_clean_scrub_stamp":"2024-09-13T16:48:36.989352+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:07.316522+0000","scrub_duration":0.0034627450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'152","reported_seq":164,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.342883+0000","last_change":"2024-09-13T16:44:58.957875+0000","last_active":"2024-09-13T16:48:22.342883+0000","last_peered":"2024-09-13T16:48:22.342883+0000","last_clean":"2024-09-13T16:48:22.342883+0000","last_became_active":"2024-09-13T16:44:58.957568+0000","last_became_peered":"2024-09-13T16:44:58.957568+0000","last_unstale":"2024-09-13T16:48:22.342883+0000","last_undegraded":"2024-09-13T16:48:22.342883+0000","last_fullsized":"2024-09-13T16:48:22.342883+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'20","reported_seq":46,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:46.886536+0000","last_change":"2024-09-13T16:48:46.886655+0000","last_active":"2024-09-13T16:48:46.886536+0000","last_peered":"2024-09-13T16:48:46.886536+0000","last_clean":"2024-09-13T16:48:46.886536+0000","last_became_active":"2024-09-13T16:43:54.559290+0000","last_became_peered":"2024-09-13T16:43:54.559290+0000","last_unstale":"2024-09-13T16:48:46.886536+0000","last_undegraded":"2024-09-13T16:48:46.886536+0000","last_fullsized":"2024-09-13T16:48:46.886536+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'20","last_scrub_stamp":"2024-09-13T16:48:46.886511+0000","last_deep_scrub":"22'20","last_deep_scrub_stamp":"2024-09-13T16:48:46.886511+0000","last_clean_scrub_stamp":"2024-09-13T16:48:46.886511+0000","objects_scrubbed":7,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:11.653937+0000","scrub_duration":0.0070396130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"22'180","reported_seq":195,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:22.351450+0000","last_change":"2024-09-13T16:43:55.000783+0000","last_active":"2024-09-13T16:48:22.351450+0000","last_peered":"2024-09-13T16:48:22.351450+0000","last_clean":"2024-09-13T16:48:22.351450+0000","last_became_active":"2024-09-13T16:43:55.000196+0000","last_became_peered":"2024-09-13T16:43:55.000196+0000","last_unstale":"2024-09-13T16:48:22.351450+0000","last_undegraded":"2024-09-13T16:48:22.351450+0000","last_fullsized":"2024-09-13T16:48:22.351450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:43:54.521415+0000","last_clean_scrub_stamp":"2024-09-13T16:43:54.521415+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":29,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:31.845619+0000","last_change":"2024-09-13T16:48:31.845740+0000","last_active":"2024-09-13T16:48:31.845619+0000","last_peered":"2024-09-13T16:48:31.845619+0000","last_clean":"2024-09-13T16:48:31.845619+0000","last_became_active":"2024-09-13T16:43:54.559586+0000","last_became_peered":"2024-09-13T16:43:54.559586+0000","last_unstale":"2024-09-13T16:48:31.845619+0000","last_undegraded":"2024-09-13T16:48:31.845619+0000","last_fullsized":"2024-09-13T16:48:31.845619+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'2","last_scrub_stamp":"2024-09-13T16:48:31.845590+0000","last_deep_scrub":"20'2","last_deep_scrub_stamp":"2024-09-13T16:48:31.845590+0000","last_clean_scrub_stamp":"2024-09-13T16:48:31.845590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:09:55.212541+0000","scrub_duration":0.0019454159999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"22'152","reported_seq":178,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:41.886790+0000","last_change":"2024-09-13T16:48:41.886897+0000","last_active":"2024-09-13T16:48:41.886790+0000","last_peered":"2024-09-13T16:48:41.886790+0000","last_clean":"2024-09-13T16:48:41.886790+0000","last_became_active":"2024-09-13T16:43:54.995802+0000","last_became_peered":"2024-09-13T16:43:54.995802+0000","last_unstale":"2024-09-13T16:48:41.886790+0000","last_undegraded":"2024-09-13T16:48:41.886790+0000","last_fullsized":"2024-09-13T16:48:41.886790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'152","last_scrub_stamp":"2024-09-13T16:48:41.886768+0000","last_deep_scrub":"22'152","last_deep_scrub_stamp":"2024-09-13T16:48:41.886768+0000","last_clean_scrub_stamp":"2024-09-13T16:48:41.886768+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:39.898801+0000","scrub_duration":0.00231227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'19","reported_seq":46,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:27.429267+0000","last_change":"2024-09-13T16:48:27.429354+0000","last_active":"2024-09-13T16:48:27.429267+0000","last_peered":"2024-09-13T16:48:27.429267+0000","last_clean":"2024-09-13T16:48:27.429267+0000","last_became_active":"2024-09-13T16:43:54.560552+0000","last_became_peered":"2024-09-13T16:43:54.560552+0000","last_unstale":"2024-09-13T16:48:27.429267+0000","last_undegraded":"2024-09-13T16:48:27.429267+0000","last_fullsized":"2024-09-13T16:48:27.429267+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'19","last_scrub_stamp":"2024-09-13T16:48:27.429247+0000","last_deep_scrub":"22'19","last_deep_scrub_stamp":"2024-09-13T16:48:27.429247+0000","last_clean_scrub_stamp":"2024-09-13T16:48:27.429247+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:55:11.474511+0000","scrub_duration":0.0049610629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11253,"ondisk_log_size":11253,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":371833,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":410,"num_write_kb":567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":743666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":465,"ondisk_log_size":465,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574910,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39628,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332212,"statfs":{"total":96636764160,"available":96596185088,"internally_reserved":0,"allocated":331776,"data_stored":675500,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":40237514},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.59199999999999997,"15min":0.57299999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":5.0620000000000003,"5min":5.0620000000000003,"15min":5.0620000000000003},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.68500000000000005,"15min":0.60099999999999998},"min":{"1min":0.372,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":5.1859999999999999,"5min":5.1859999999999999,"15min":5.1859999999999999},"last":0.61599999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.36799999999999999,"5min":0.314,"15min":0.314},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.61699999999999999,"15min":0.58999999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":3.6070000000000002,"5min":3.6070000000000002,"15min":3.6070000000000002},"last":0.57399999999999995}]},{"osd":2,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":2.4929999999999999,"5min":2.4929999999999999,"15min":2.4929999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.63400000000000001,"15min":0.61599999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":0.66400000000000003}]},{"osd":3,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.56899999999999995,"15min":0.58199999999999996},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.8490000000000002,"5min":3.8490000000000002,"15min":3.8490000000000002},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64800000000000002,"15min":0.60499999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.3160000000000001,"5min":4.0129999999999999,"15min":4.0129999999999999},"last":0.64800000000000002}]},{"osd":4,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.434,"15min":0.442},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.78000000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45000000000000001,"15min":0.49199999999999999},"min":{"1min":0.27200000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67300000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.39800000000000002}]},{"osd":5,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.46300000000000002,"15min":0.48699999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.38},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.499,"15min":0.52200000000000002},"min":{"1min":0.32600000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.753,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.45100000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.53300000000000003,"15min":0.56299999999999994},"min":{"1min":0.36399999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81100000000000005,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.504,"15min":0.54000000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72799999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.46300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574910,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35812,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336028,"statfs":{"total":96636764160,"available":96600092672,"internally_reserved":0,"allocated":618496,"data_stored":448920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36043209},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":0.69499999999999995,"15min":0.625},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":8.468,"5min":8.468,"15min":8.468},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.67300000000000004,"15min":0.625},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":7.5739999999999998,"5min":7.5739999999999998,"15min":7.5739999999999998},"last":0.82099999999999995}]},{"osd":1,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.29299999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60799999999999998,"15min":0.63900000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.79500000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.028,"5min":2.431,"15min":2.431},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.466,"5min":0.78500000000000003,"15min":0.64100000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.9139999999999997,"5min":6.9139999999999997,"15min":6.9139999999999997},"last":0.72799999999999998}]},{"osd":3,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":0.67200000000000004,"15min":0.60999999999999999},"min":{"1min":0.373,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.2400000000000002,"5min":6.2400000000000002,"15min":6.2400000000000002},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.70899999999999996,"15min":0.66200000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.4500000000000002,"5min":3.4500000000000002,"15min":3.4500000000000002},"last":0.76300000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.41299999999999998,"15min":0.45100000000000001},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.54300000000000004,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.376},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.439,"15min":0.48299999999999998},"min":{"1min":0.23699999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.39500000000000002}]},{"osd":5,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.48799999999999999,"15min":0.52800000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.435},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.51100000000000001,"15min":0.56499999999999995},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.73599999999999999,"5min":0.752,"15min":0.752},"last":0.49299999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:47:55 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.48199999999999998,"15min":0.52000000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.71299999999999997,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.54000000000000004,"15min":0.58099999999999996},"min":{"1min":0.34000000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.45800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574909,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39572,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332268,"statfs":{"total":96636764160,"available":96596242432,"internally_reserved":0,"allocated":274432,"data_stored":86302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.58299999999999996,"15min":0.56299999999999994},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.411,"5min":3.411,"15min":3.411},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.117,"5min":3.117,"15min":3.117},"last":0.61499999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55500000000000005,"15min":0.56100000000000005},"min":{"1min":0.29899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.64300000000000002,"15min":0.58899999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.843,"5min":4.3860000000000001,"15min":4.3860000000000001},"last":0.60399999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.67300000000000004,"15min":0.64300000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":4.0819999999999999,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.64700000000000002,"15min":0.61399999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":4.3170000000000002,"5min":4.3170000000000002,"15min":4.3170000000000002},"last":0.63600000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.90700000000000003,"15min":0.68700000000000006},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.4690000000000003,"5min":28.707000000000001,"15min":28.707000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.68700000000000006,"15min":0.625},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.2599999999999998,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":0.65000000000000002}]},{"osd":4,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.41899999999999998,"15min":0.45100000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.64700000000000002,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.45500000000000002,"15min":0.48099999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.70399999999999996,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.48999999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.441,"15min":0.45700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.67000000000000004,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.46500000000000002,"15min":0.48499999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.54800000000000004}]},{"osd":7,"last update":"Fri Sep 13 16:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.49099999999999999,"15min":0.52000000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65600000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.501,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.56499999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39552,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332288,"statfs":{"total":96636764160,"available":96596262912,"internally_reserved":0,"allocated":253952,"data_stored":84762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":3.7160000000000002,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.61099999999999999,"15min":0.59399999999999997},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.448,"5min":2.9980000000000002,"15min":2.9980000000000002},"last":0.78300000000000003}]},{"osd":1,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.69599999999999995,"15min":0.63},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.3700000000000001,"5min":4.3700000000000001,"15min":4.3700000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.63500000000000001,"15min":0.61499999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.9020000000000001,"5min":3.9020000000000001,"15min":3.9020000000000001},"last":0.82899999999999996}]},{"osd":2,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":2.085,"5min":2.613,"15min":2.613},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.194,"5min":0.72499999999999998,"15min":0.65800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":7.9809999999999999,"5min":7.9809999999999999,"15min":7.9809999999999999},"last":0.81100000000000005}]},{"osd":3,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.65800000000000003,"15min":0.63400000000000001},"min":{"1min":0.36199999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.68600000000000005,"15min":0.68899999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.8130000000000002,"5min":2.8130000000000002,"15min":2.8130000000000002},"last":0.73099999999999998}]},{"osd":5,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.42799999999999999,"15min":0.46100000000000002},"min":{"1min":0.25600000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.438,"15min":0.47299999999999998},"min":{"1min":0.254,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61399999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.47299999999999998}]},{"osd":6,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.442,"15min":0.47999999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.628,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.48599999999999999,"15min":0.53400000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.63600000000000001,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.45200000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.48099999999999998,"15min":0.50900000000000001},"min":{"1min":0.308,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.436},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.54400000000000004,"15min":0.60399999999999998},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70499999999999996,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.48799999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574910,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44012,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94327828,"statfs":{"total":96636764160,"available":96591695872,"internally_reserved":0,"allocated":626688,"data_stored":449059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":44431816},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.57899999999999996,"15min":0.49199999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.8449999999999998,"5min":4.8449999999999998,"15min":4.8449999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.5529999999999999,"5min":0.77800000000000002,"15min":0.59799999999999998},"min":{"1min":0.41099999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":7.7489999999999997,"5min":7.7489999999999997,"15min":7.7489999999999997},"last":0.627}]},{"osd":1,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":2.0529999999999999,"5min":0.84299999999999997,"15min":0.60499999999999998},"min":{"1min":0.317,"5min":0.252,"15min":0.252},"max":{"1min":28.908999999999999,"5min":28.908999999999999,"15min":28.908999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":0.73299999999999998,"15min":0.64600000000000002},"min":{"1min":0.40600000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.1520000000000001,"5min":3.1699999999999999,"15min":3.1699999999999999},"last":0.47499999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":0.80300000000000005,"15min":0.63500000000000001},"min":{"1min":0.41799999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":4.3630000000000004,"5min":4.3630000000000004,"15min":4.3630000000000004},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":0.76900000000000002,"15min":0.69299999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.1859999999999999,"5min":3.2610000000000001,"15min":3.2610000000000001},"last":0.53300000000000003}]},{"osd":4,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":0.94299999999999995,"15min":0.82699999999999996},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":4.1029999999999998,"5min":4.1029999999999998,"15min":4.1029999999999998},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.79300000000000004,"15min":0.753},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.903,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":0.753}]},{"osd":5,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":0.80700000000000005,"15min":0.754},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":2.012,"5min":2.5790000000000002,"15min":2.5790000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.1160000000000001,"5min":0.89100000000000001,"15min":0.84699999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.0739999999999998,"5min":3.1970000000000001,"15min":3.1970000000000001},"last":0.72799999999999998}]},{"osd":6,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":0.96699999999999997,"15min":0.84199999999999997},"min":{"1min":0.61099999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":4.1449999999999996,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.0760000000000001,"5min":0.86499999999999999,"15min":0.82799999999999996},"min":{"1min":0.53000000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.9319999999999999,"5min":2.7360000000000002,"15min":2.7360000000000002},"last":0.78400000000000003}]},{"osd":7,"last update":"Fri Sep 13 16:47:56 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":0.88500000000000001,"15min":0.82399999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":3.117,"5min":3.2320000000000002,"15min":3.2320000000000002},"last":0.67000000000000004},{"interface":"front","average":{"1min":1.613,"5min":1.006,"15min":0.874},"min":{"1min":0.56599999999999995,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":4.181,"5min":4.181,"15min":4.181},"last":0.89600000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574910,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35464,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336376,"statfs":{"total":96636764160,"available":96600449024,"internally_reserved":0,"allocated":262144,"data_stored":84385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.988,"5min":0.996,"15min":0.65800000000000003},"min":{"1min":0.26600000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":31.440999999999999,"5min":31.440999999999999,"15min":31.440999999999999},"last":0.37},{"interface":"front","average":{"1min":1.294,"5min":0.69099999999999995,"15min":0.57699999999999996},"min":{"1min":0.34899999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.7720000000000002,"5min":4.7720000000000002,"15min":4.7720000000000002},"last":0.44500000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.2949999999999999,"5min":0.89800000000000002,"15min":0.65600000000000003},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":8.7780000000000005,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.121,"5min":0.755,"15min":0.61299999999999999},"min":{"1min":0.379,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.0230000000000001,"5min":6.5659999999999998,"15min":6.5659999999999998},"last":0.51400000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":0.71499999999999997,"15min":0.59299999999999997},"min":{"1min":0.371,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":6.5469999999999997,"5min":6.5469999999999997,"15min":6.5469999999999997},"last":0.626},{"interface":"front","average":{"1min":1.7869999999999999,"5min":0.84499999999999997,"15min":0.68300000000000005},"min":{"1min":0.45500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":8.8659999999999997,"5min":8.8659999999999997,"15min":8.8659999999999997},"last":0.69599999999999995}]},{"osd":4,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":0.81499999999999995,"15min":0.752},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":3.4460000000000002,"5min":3.4460000000000002,"15min":3.4460000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.163,"5min":0.80600000000000005,"15min":0.76300000000000001},"min":{"1min":0.46200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":6.4379999999999997,"5min":6.4379999999999997,"15min":6.4379999999999997},"last":0.80600000000000005}]},{"osd":5,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.929,"5min":0.97599999999999998,"15min":0.81699999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":9.1039999999999992,"5min":9.1039999999999992,"15min":9.1039999999999992},"last":0.751},{"interface":"front","average":{"1min":1.1899999999999999,"5min":0.84399999999999997,"15min":0.77400000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":3.633,"5min":3.633,"15min":3.633},"last":0.71899999999999997}]},{"osd":6,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":0.83199999999999996,"15min":0.79400000000000004},"min":{"1min":0.54400000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":6.5140000000000002,"5min":6.5140000000000002,"15min":6.5140000000000002},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.95,"5min":0.999,"15min":0.85899999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":9.141,"5min":9.141,"15min":9.141},"last":0.79600000000000004}]},{"osd":7,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":0.88400000000000001,"15min":0.84299999999999997},"min":{"1min":0.50700000000000001,"5min":0.505,"15min":0.505},"max":{"1min":3.5990000000000002,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.2450000000000001,"5min":0.86699999999999999,"15min":0.83499999999999996},"min":{"1min":0.57599999999999996,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":6.4800000000000004,"5min":6.4800000000000004,"15min":6.4800000000000004},"last":0.97199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35460,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336380,"statfs":{"total":96636764160,"available":96600453120,"internally_reserved":0,"allocated":258048,"data_stored":84564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5630000000000002,"5min":1.0009999999999999,"15min":0.65200000000000002},"min":{"1min":0.29399999999999998,"5min":0.249,"15min":0.249},"max":{"1min":22.721,"5min":22.721,"15min":22.721},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.226,"5min":0.86699999999999999,"15min":0.64100000000000001},"min":{"1min":0.32200000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.2430000000000003,"5min":19.494,"15min":19.494},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3570000000000002,"5min":0.98999999999999999,"15min":0.67200000000000004},"min":{"1min":0.34799999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":15.491,"5min":15.491,"15min":15.491},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.5549999999999999,"5min":0.93300000000000005,"15min":0.70399999999999996},"min":{"1min":0.39600000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.6829999999999998,"5min":17.071000000000002,"15min":17.071000000000002},"last":0.44700000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.631,"5min":0.98599999999999999,"15min":0.73499999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":6.883,"5min":18.838000000000001,"15min":18.838000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.339,"5min":0.92200000000000004,"15min":0.69699999999999995},"min":{"1min":0.40899999999999997,"5min":0.31,"15min":0.31},"max":{"1min":7.0339999999999998,"5min":20.199000000000002,"15min":20.199000000000002},"last":0.56200000000000006}]},{"osd":4,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":2.778,"5min":1.2390000000000001,"15min":0.91300000000000003},"min":{"1min":0.39800000000000002,"5min":0.373,"15min":0.373},"max":{"1min":15.654999999999999,"5min":15.654999999999999,"15min":15.654999999999999},"last":0.745},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.091,"15min":0.89500000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":7.5659999999999998,"5min":18.742000000000001,"15min":18.742000000000001},"last":0.72399999999999998}]},{"osd":5,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":0.97699999999999998,"15min":0.89300000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":7.3230000000000004,"5min":7.3230000000000004,"15min":7.3230000000000004},"last":0.76500000000000001},{"interface":"front","average":{"1min":2.819,"5min":1.296,"15min":0.98299999999999998},"min":{"1min":0.51400000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":15.686,"5min":15.686,"15min":15.686},"last":0.71099999999999997}]},{"osd":6,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":0.96799999999999997,"15min":0.86599999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":7.3700000000000001,"5min":7.3700000000000001,"15min":7.3700000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":2.851,"5min":1.3180000000000001,"15min":1.002},"min":{"1min":0.55500000000000005,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":15.718,"5min":15.718,"15min":15.718},"last":0.81499999999999995}]},{"osd":7,"last update":"Fri Sep 13 16:47:58 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":1.119,"15min":0.89500000000000002},"min":{"1min":0.495,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":8.1539999999999999,"5min":18.798999999999999,"15min":18.798999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.4339999999999999,"5min":0.995,"15min":0.90200000000000002},"min":{"1min":0.53000000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":7.407,"5min":7.4950000000000001,"15min":7.4950000000000001},"last":0.67800000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43712,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94328128,"statfs":{"total":96636764160,"available":96592003072,"internally_reserved":0,"allocated":319488,"data_stored":673303,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":44431814},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.65800000000000003,"15min":0.53000000000000003},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.9239999999999999,"5min":7.2539999999999996,"15min":7.2539999999999996},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.2589999999999999,"5min":0.71999999999999997,"15min":0.55700000000000005},"min":{"1min":0.32400000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":6.0949999999999998,"5min":6.0949999999999998,"15min":6.0949999999999998},"last":0.65400000000000003}]},{"osd":2,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.78300000000000003,"15min":0.627},"min":{"1min":0.41899999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.1800000000000002,"5min":7.1920000000000002,"15min":7.1920000000000002},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.151,"5min":0.74299999999999999,"15min":0.59899999999999998},"min":{"1min":0.36399999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":6.5,"5min":6.5,"15min":6.5},"last":0.72499999999999998}]},{"osd":3,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":0.66500000000000004,"15min":0.54800000000000004},"min":{"1min":0.38800000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":4.4459999999999997,"5min":4.4459999999999997,"15min":4.4459999999999997},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.85999999999999999,"15min":0.69099999999999995},"min":{"1min":0.38700000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0409999999999999,"5min":10.522,"15min":10.522},"last":0.84999999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":0.89300000000000002,"15min":0.80100000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":6.4560000000000004,"5min":6.4560000000000004,"15min":6.4560000000000004},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.82799999999999996,"15min":0.77900000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":3.79,"5min":3.79,"15min":3.79},"last":0.78800000000000003}]},{"osd":5,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":0.85399999999999998,"15min":0.78800000000000003},"min":{"1min":0.51300000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.8340000000000001,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.002,"5min":0.96599999999999997,"15min":0.85399999999999998},"min":{"1min":0.48599999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":4.5620000000000003,"5min":7.8339999999999996,"15min":7.8339999999999996},"last":0.91200000000000003}]},{"osd":6,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.98199999999999998,"15min":0.86499999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.7270000000000003,"5min":7.5510000000000002,"15min":7.5510000000000002},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.345,"5min":0.95699999999999996,"15min":0.81499999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":7.444,"5min":7.444,"15min":7.444},"last":0.80400000000000005}]},{"osd":7,"last update":"Fri Sep 13 16:47:57 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":0.95399999999999996,"15min":0.81599999999999995},"min":{"1min":0.51900000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":7.5350000000000001,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.1639999999999999,"5min":0.874,"15min":0.79700000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.1879999999999997,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.96299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":365979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":389120,"data_stored":365840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:48:50.816 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.816 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.816 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.816 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.817 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.817 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.817 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.817 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.817 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.817 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.818 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.818 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.818 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.818 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.818 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.818 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.818 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.819 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.819 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.819 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.819 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.819 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.820 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.820 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.820 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.820 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.820 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.821 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.821 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.821 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.821 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-09-13T16:43:54.521415+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=43, tm_sec=54, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=48, tm_sec=24, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:48:50.821 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T16:49:04.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:49:04.791 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:49:10.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:49:10.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.986+0000 7f50ea600640 1 Processor -- start 2024-09-13T16:49:10.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.986+0000 7f50ea600640 1 -- start start 2024-09-13T16:49:10.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.987+0000 7f50ea600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50e4150ca0 0x7f50e41568e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:10.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.987+0000 7f50ea600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f50e40c2250 0x7f50e415a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:10.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.987+0000 7f50ea600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e4151080 con 0x7f50e40c4d60 2024-09-13T16:49:10.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.987+0000 7f50ea600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e40c2670 con 0x7f50e40c2250 2024-09-13T16:49:10.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.987+0000 7f50ea600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e406a990 con 0x7f50e4150ca0 2024-09-13T16:49:10.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.988+0000 7f50daa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f50e40c2250 0x7f50e415a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:10.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.988+0000 7f50daa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f50e40c2250 0x7f50e415a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35620/0 (socket says 172.21.15.28:35620) 2024-09-13T16:49:10.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.988+0000 7f50e3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f50e40c4d60 0x7f50e406d8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55242/0 (socket says 172.21.15.28:55242) 2024-09-13T16:49:10.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.988+0000 7f50daa00640 1 -- 172.21.15.28:0/1987858288 learned_addr learned my addr 172.21.15.28:0/1987858288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:49:10.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.988+0000 7f50e3400640 1 --2- 172.21.15.28:0/1987858288 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50e4150ca0 0x7f50e41568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:10.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.988+0000 7f50daa00640 1 -- 172.21.15.28:0/1987858288 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50e4150ca0 msgr2=0x7f50e41568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:49:10.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.988+0000 7f50daa00640 1 --2- 172.21.15.28:0/1987858288 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50e4150ca0 0x7f50e41568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:10.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.988+0000 7f50daa00640 1 -- 172.21.15.28:0/1987858288 >> v1:172.21.15.28:6789/0 conn(0x7f50e40c4d60 legacy=0x7f50e406d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:10.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.988+0000 7f50e2a00640 1 -- 172.21.15.28:0/1987858288 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658816281 0 0) 0x7f50e4151080 con 0x7f50e40c4d60 2024-09-13T16:49:10.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.988+0000 7f50daa00640 1 -- 172.21.15.28:0/1987858288 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e40c2870 con 0x7f50e40c2250 2024-09-13T16:49:10.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.989+0000 7f50daa00640 1 --2- 172.21.15.28:0/1987858288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50e40c2250 0x7f50e415a730 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009870 tx=0x7f50cc02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be04ef6f73fe8b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:10.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.990+0000 7f50e2a00640 1 -- 172.21.15.28:0/1987858288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc032440 con 0x7f50e40c2250 2024-09-13T16:49:10.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.990+0000 7f50e2a00640 1 -- 172.21.15.28:0/1987858288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50cc0325a0 con 0x7f50e40c2250 2024-09-13T16:49:10.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.990+0000 7f50e2a00640 1 -- 172.21.15.28:0/1987858288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc032930 con 0x7f50e40c2250 2024-09-13T16:49:10.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.990+0000 7f50ea600640 1 -- 172.21.15.28:0/1987858288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50e40c2250 msgr2=0x7f50e415a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:10.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.990+0000 7f50ea600640 1 --2- 172.21.15.28:0/1987858288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50e40c2250 0x7f50e415a730 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009870 tx=0x7f50cc02a1c0 comp rx=0 tx=0).stop 2024-09-13T16:49:10.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.990+0000 7f50ea600640 1 -- 172.21.15.28:0/1987858288 shutdown_connections 2024-09-13T16:49:10.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.990+0000 7f50ea600640 1 --2- 172.21.15.28:0/1987858288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50e40c2250 0x7f50e415a730 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:10.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.990+0000 7f50ea600640 1 --2- 172.21.15.28:0/1987858288 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50e4150ca0 0x7f50e41568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:10.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.990+0000 7f50ea600640 1 -- 172.21.15.28:0/1987858288 >> 172.21.15.28:0/1987858288 conn(0x7f50e4066040 msgr2=0x7f50e4069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:10.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.990+0000 7f50ea600640 1 -- 172.21.15.28:0/1987858288 shutdown_connections 2024-09-13T16:49:10.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.990+0000 7f50ea600640 1 -- 172.21.15.28:0/1987858288 wait complete. 2024-09-13T16:49:10.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.992+0000 7f50ea600640 1 Processor -- start 2024-09-13T16:49:10.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.992+0000 7f50ea600640 1 -- start start 2024-09-13T16:49:10.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.993+0000 7f50ea600640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50e40c2250 0x7f50e4159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:10.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.993+0000 7f50ea600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50e40c4d60 0x7f50e4070b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:10.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.993+0000 7f50ea600640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50e4150ca0 0x7f50e4155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:10.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.993+0000 7f50ea600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e4070230 con 0x7f50e40c4d60 2024-09-13T16:49:10.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.993+0000 7f50ea600640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e40c10c0 con 0x7f50e4150ca0 2024-09-13T16:49:10.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.993+0000 7f50ea600640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e406fef0 con 0x7f50e40c2250 2024-09-13T16:49:10.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.993+0000 7f50daa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50e40c4d60 0x7f50e4070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:10.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.993+0000 7f50daa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50e40c4d60 0x7f50e4070b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35624/0 (socket says 172.21.15.28:35624) 2024-09-13T16:49:10.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.993+0000 7f50e3e00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50e4150ca0 0x7f50e4155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:10.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.993+0000 7f50daa00640 1 -- 172.21.15.28:0/2919370814 learned_addr learned my addr 172.21.15.28:0/2919370814 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:49:10.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.993+0000 7f50e3400640 1 --2- 172.21.15.28:0/2919370814 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50e40c2250 0x7f50e4159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:10.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.994+0000 7f50daa00640 1 -- 172.21.15.28:0/2919370814 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50e40c2250 msgr2=0x7f50e4159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:10.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.994+0000 7f50daa00640 1 --2- 172.21.15.28:0/2919370814 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50e40c2250 0x7f50e4159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:10.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.994+0000 7f50daa00640 1 -- 172.21.15.28:0/2919370814 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50e4150ca0 msgr2=0x7f50e4155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:10.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.994+0000 7f50daa00640 1 --2- 172.21.15.28:0/2919370814 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50e4150ca0 0x7f50e4155660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:10.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.994+0000 7f50daa00640 1 -- 172.21.15.28:0/2919370814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc0094d0 con 0x7f50e40c4d60 2024-09-13T16:49:10.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.994+0000 7f50e3400640 1 --2- 172.21.15.28:0/2919370814 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50e40c2250 0x7f50e4159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:49:10.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.994+0000 7f50daa00640 1 --2- 172.21.15.28:0/2919370814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50e40c4d60 0x7f50e4070b80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009990 tx=0x7f50cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:10.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.994+0000 7f50dbe00640 1 -- 172.21.15.28:0/2919370814 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc032440 con 0x7f50e40c4d60 2024-09-13T16:49:10.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.994+0000 7f50dbe00640 1 -- 172.21.15.28:0/2919370814 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50cc004230 con 0x7f50e40c4d60 2024-09-13T16:49:10.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.994+0000 7f50dbe00640 1 -- 172.21.15.28:0/2919370814 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc032e20 con 0x7f50e40c4d60 2024-09-13T16:49:10.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.995+0000 7f50ea600640 1 -- 172.21.15.28:0/2919370814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e40c0c30 con 0x7f50e40c4d60 2024-09-13T16:49:10.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.995+0000 7f50ea600640 1 -- 172.21.15.28:0/2919370814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e414fce0 con 0x7f50e40c4d60 2024-09-13T16:49:10.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.996+0000 7f50ea600640 1 -- 172.21.15.28:0/2919370814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e40c3110 con 0x7f50e40c4d60 2024-09-13T16:49:10.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.997+0000 7f50dbe00640 1 -- 172.21.15.28:0/2919370814 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f50cc030980 con 0x7f50e40c4d60 2024-09-13T16:49:10.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.997+0000 7f50dbe00640 1 --2- 172.21.15.28:0/2919370814 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f50c006ac00 0x7f50c006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:10.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.997+0000 7f50dbe00640 1 -- 172.21.15.28:0/2919370814 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f50cc0aa430 con 0x7f50e40c4d60 2024-09-13T16:49:10.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.998+0000 7f50e3400640 1 --2- 172.21.15.28:0/2919370814 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f50c006ac00 0x7f50c006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:10.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:10.998+0000 7f50e3400640 1 --2- 172.21.15.28:0/2919370814 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f50c006ac00 0x7f50c006d0c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009e90 tx=0x7f50d4009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:11.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.002+0000 7f50dbe00640 1 -- 172.21.15.28:0/2919370814 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f50cc075760 con 0x7f50e40c4d60 2024-09-13T16:49:11.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.176+0000 7f50ea600640 1 -- 172.21.15.28:0/2919370814 --> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f50e40c7250 con 0x7f50c006ac00 2024-09-13T16:49:11.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.185+0000 7f50dbe00640 1 -- 172.21.15.28:0/2919370814 <== mgr.4109 v2:172.21.15.191:6832/1568228941 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+292686 (secure 0 0 0) 0x7f50e40c7250 con 0x7f50c006ac00 2024-09-13T16:49:11.190 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:49:11.192 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T16:49:11.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.189+0000 7f50ea600640 1 -- 172.21.15.28:0/2919370814 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f50c006ac00 msgr2=0x7f50c006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:11.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.189+0000 7f50ea600640 1 --2- 172.21.15.28:0/2919370814 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f50c006ac00 0x7f50c006d0c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009e90 tx=0x7f50d4009190 comp rx=0 tx=0).stop 2024-09-13T16:49:11.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.189+0000 7f50ea600640 1 -- 172.21.15.28:0/2919370814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50e40c4d60 msgr2=0x7f50e4070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:11.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.189+0000 7f50ea600640 1 --2- 172.21.15.28:0/2919370814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50e40c4d60 0x7f50e4070b80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009990 tx=0x7f50cc0046c0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.189+0000 7f50ea600640 1 -- 172.21.15.28:0/2919370814 shutdown_connections 2024-09-13T16:49:11.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.189+0000 7f50ea600640 1 --2- 172.21.15.28:0/2919370814 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f50c006ac00 0x7f50c006d0c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.189+0000 7f50ea600640 1 --2- 172.21.15.28:0/2919370814 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50e4150ca0 0x7f50e4155660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.189+0000 7f50ea600640 1 --2- 172.21.15.28:0/2919370814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50e40c4d60 0x7f50e4070b80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.189+0000 7f50ea600640 1 --2- 172.21.15.28:0/2919370814 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50e40c2250 0x7f50e4159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.189+0000 7f50ea600640 1 -- 172.21.15.28:0/2919370814 >> 172.21.15.28:0/2919370814 conn(0x7f50e4066040 msgr2=0x7f50e40c13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:11.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.189+0000 7f50ea600640 1 -- 172.21.15.28:0/2919370814 shutdown_connections 2024-09-13T16:49:11.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.189+0000 7f50ea600640 1 -- 172.21.15.28:0/2919370814 wait complete. 2024-09-13T16:49:11.207 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-09-13T16:49:09.664927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":962201,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":121,"num_read_kb":105,"num_write":9785,"num_write_kb":25168807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11837,"ondisk_log_size":11837,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":313212,"kb_used_data":2876,"kb_used_omap":12,"kb_used_meta":310259,"kb_avail":754661508,"statfs":{"total":773094113280,"available":772773384192,"internally_reserved":0,"allocated":2945024,"data_stored":2586795,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12743,"internal_metadata":317705785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003203"},"pg_stats":[{"pgid":"3.3f","version":"22'187","reported_seq":213,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:49:03.896411+0000","last_change":"2024-09-13T16:49:03.896524+0000","last_active":"2024-09-13T16:49:03.896411+0000","last_peered":"2024-09-13T16:49:03.896411+0000","last_clean":"2024-09-13T16:49:03.896411+0000","last_became_active":"2024-09-13T16:43:54.986988+0000","last_became_peered":"2024-09-13T16:43:54.986988+0000","last_unstale":"2024-09-13T16:49:03.896411+0000","last_undegraded":"2024-09-13T16:49:03.896411+0000","last_fullsized":"2024-09-13T16:49:03.896411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'187","last_scrub_stamp":"2024-09-13T16:49:03.896388+0000","last_deep_scrub":"22'187","last_deep_scrub_stamp":"2024-09-13T16:49:03.896388+0000","last_clean_scrub_stamp":"2024-09-13T16:49:03.896388+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:41.699574+0000","scrub_duration":0.0014872749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"22'138","reported_seq":164,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:53.041461+0000","last_change":"2024-09-13T16:48:53.041562+0000","last_active":"2024-09-13T16:48:53.041461+0000","last_peered":"2024-09-13T16:48:53.041461+0000","last_clean":"2024-09-13T16:48:53.041461+0000","last_became_active":"2024-09-13T16:43:55.000757+0000","last_became_peered":"2024-09-13T16:43:55.000757+0000","last_unstale":"2024-09-13T16:48:53.041461+0000","last_undegraded":"2024-09-13T16:48:53.041461+0000","last_fullsized":"2024-09-13T16:48:53.041461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'138","last_scrub_stamp":"2024-09-13T16:48:53.041440+0000","last_deep_scrub":"22'138","last_deep_scrub_stamp":"2024-09-13T16:48:53.041440+0000","last_clean_scrub_stamp":"2024-09-13T16:48:53.041440+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:05.483373+0000","scrub_duration":0.0018234740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"22'212","reported_seq":238,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:35.441355+0000","last_change":"2024-09-13T16:48:35.441475+0000","last_active":"2024-09-13T16:48:35.441355+0000","last_peered":"2024-09-13T16:48:35.441355+0000","last_clean":"2024-09-13T16:48:35.441355+0000","last_became_active":"2024-09-13T16:43:54.986198+0000","last_became_peered":"2024-09-13T16:43:54.986198+0000","last_unstale":"2024-09-13T16:48:35.441355+0000","last_undegraded":"2024-09-13T16:48:35.441355+0000","last_fullsized":"2024-09-13T16:48:35.441355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'212","last_scrub_stamp":"2024-09-13T16:48:35.441330+0000","last_deep_scrub":"22'212","last_deep_scrub_stamp":"2024-09-13T16:48:35.441330+0000","last_clean_scrub_stamp":"2024-09-13T16:48:35.441330+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:44:17.520802+0000","scrub_duration":0.0017448349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"22'133","reported_seq":159,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:54.631851+0000","last_change":"2024-09-13T16:48:54.631942+0000","last_active":"2024-09-13T16:48:54.631851+0000","last_peered":"2024-09-13T16:48:54.631851+0000","last_clean":"2024-09-13T16:48:54.631851+0000","last_became_active":"2024-09-13T16:43:54.991804+0000","last_became_peered":"2024-09-13T16:43:54.991804+0000","last_unstale":"2024-09-13T16:48:54.631851+0000","last_undegraded":"2024-09-13T16:48:54.631851+0000","last_fullsized":"2024-09-13T16:48:54.631851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'133","last_scrub_stamp":"2024-09-13T16:48:54.631829+0000","last_deep_scrub":"22'133","last_deep_scrub_stamp":"2024-09-13T16:48:54.631829+0000","last_clean_scrub_stamp":"2024-09-13T16:48:54.631829+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:31.970216+0000","scrub_duration":0.0018563379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":270336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"22'194","reported_seq":220,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:49:04.874237+0000","last_change":"2024-09-13T16:49:04.875045+0000","last_active":"2024-09-13T16:49:04.874237+0000","last_peered":"2024-09-13T16:49:04.874237+0000","last_clean":"2024-09-13T16:49:04.874237+0000","last_became_active":"2024-09-13T16:43:54.987328+0000","last_became_peered":"2024-09-13T16:43:54.987328+0000","last_unstale":"2024-09-13T16:49:04.874237+0000","last_undegraded":"2024-09-13T16:49:04.874237+0000","last_fullsized":"2024-09-13T16:49:04.874237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'194","last_scrub_stamp":"2024-09-13T16:49:04.874196+0000","last_deep_scrub":"22'194","last_deep_scrub_stamp":"2024-09-13T16:49:04.874196+0000","last_clean_scrub_stamp":"2024-09-13T16:49:04.874196+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:30:03.572661+0000","scrub_duration":0.0018423669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"22'156","reported_seq":182,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:49:03.624760+0000","last_change":"2024-09-13T16:49:03.624858+0000","last_active":"2024-09-13T16:49:03.624760+0000","last_peered":"2024-09-13T16:49:03.624760+0000","last_clean":"2024-09-13T16:49:03.624760+0000","last_became_active":"2024-09-13T16:43:54.999752+0000","last_became_peered":"2024-09-13T16:43:54.999752+0000","last_unstale":"2024-09-13T16:49:03.624760+0000","last_undegraded":"2024-09-13T16:49:03.624760+0000","last_fullsized":"2024-09-13T16:49:03.624760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'156","last_scrub_stamp":"2024-09-13T16:49:03.624735+0000","last_deep_scrub":"22'156","last_deep_scrub_stamp":"2024-09-13T16:49:03.624735+0000","last_clean_scrub_stamp":"2024-09-13T16:49:03.624735+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:30:02.323200+0000","scrub_duration":0.0021636789999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"22'175","reported_seq":201,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:49.950503+0000","last_change":"2024-09-13T16:48:49.950503+0000","last_active":"2024-09-13T16:48:49.950503+0000","last_peered":"2024-09-13T16:48:49.950503+0000","last_clean":"2024-09-13T16:48:49.950503+0000","last_became_active":"2024-09-13T16:43:54.999350+0000","last_became_peered":"2024-09-13T16:43:54.999350+0000","last_unstale":"2024-09-13T16:48:49.950503+0000","last_undegraded":"2024-09-13T16:48:49.950503+0000","last_fullsized":"2024-09-13T16:48:49.950503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'175","last_scrub_stamp":"2024-09-13T16:48:49.950345+0000","last_deep_scrub":"22'175","last_deep_scrub_stamp":"2024-09-13T16:48:49.950345+0000","last_clean_scrub_stamp":"2024-09-13T16:48:49.950345+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:21.431046+0000","scrub_duration":0.0022390380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"22'165","reported_seq":191,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:57.993859+0000","last_change":"2024-09-13T16:48:57.993859+0000","last_active":"2024-09-13T16:48:57.993859+0000","last_peered":"2024-09-13T16:48:57.993859+0000","last_clean":"2024-09-13T16:48:57.993859+0000","last_became_active":"2024-09-13T16:43:54.996309+0000","last_became_peered":"2024-09-13T16:43:54.996309+0000","last_unstale":"2024-09-13T16:48:57.993859+0000","last_undegraded":"2024-09-13T16:48:57.993859+0000","last_fullsized":"2024-09-13T16:48:57.993859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'165","last_scrub_stamp":"2024-09-13T16:48:57.993700+0000","last_deep_scrub":"22'165","last_deep_scrub_stamp":"2024-09-13T16:48:57.993700+0000","last_clean_scrub_stamp":"2024-09-13T16:48:57.993700+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:35.332087+0000","scrub_duration":0.0024181739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"22'155","reported_seq":181,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:49:02.922362+0000","last_change":"2024-09-13T16:49:02.922460+0000","last_active":"2024-09-13T16:49:02.922362+0000","last_peered":"2024-09-13T16:49:02.922362+0000","last_clean":"2024-09-13T16:49:02.922362+0000","last_became_active":"2024-09-13T16:43:54.993475+0000","last_became_peered":"2024-09-13T16:43:54.993475+0000","last_unstale":"2024-09-13T16:49:02.922362+0000","last_undegraded":"2024-09-13T16:49:02.922362+0000","last_fullsized":"2024-09-13T16:49:02.922362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'155","last_scrub_stamp":"2024-09-13T16:49:02.922340+0000","last_deep_scrub":"22'155","last_deep_scrub_stamp":"2024-09-13T16:49:02.922340+0000","last_clean_scrub_stamp":"2024-09-13T16:49:02.922340+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:09.244826+0000","scrub_duration":0.00147371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"22'189","reported_seq":215,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:56.607944+0000","last_change":"2024-09-13T16:48:56.608055+0000","last_active":"2024-09-13T16:48:56.607944+0000","last_peered":"2024-09-13T16:48:56.607944+0000","last_clean":"2024-09-13T16:48:56.607944+0000","last_became_active":"2024-09-13T16:43:54.985979+0000","last_became_peered":"2024-09-13T16:43:54.985979+0000","last_unstale":"2024-09-13T16:48:56.607944+0000","last_undegraded":"2024-09-13T16:48:56.607944+0000","last_fullsized":"2024-09-13T16:48:56.607944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'189","last_scrub_stamp":"2024-09-13T16:48:56.607922+0000","last_deep_scrub":"22'189","last_deep_scrub_stamp":"2024-09-13T16:48:56.607922+0000","last_clean_scrub_stamp":"2024-09-13T16:48:56.607922+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:28.088623+0000","scrub_duration":0.0015724370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"22'174","reported_seq":200,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:42.717053+0000","last_change":"2024-09-13T16:48:42.717053+0000","last_active":"2024-09-13T16:48:42.717053+0000","last_peered":"2024-09-13T16:48:42.717053+0000","last_clean":"2024-09-13T16:48:42.717053+0000","last_became_active":"2024-09-13T16:43:54.985948+0000","last_became_peered":"2024-09-13T16:43:54.985948+0000","last_unstale":"2024-09-13T16:48:42.717053+0000","last_undegraded":"2024-09-13T16:48:42.717053+0000","last_fullsized":"2024-09-13T16:48:42.717053+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'174","last_scrub_stamp":"2024-09-13T16:48:42.716932+0000","last_deep_scrub":"22'174","last_deep_scrub_stamp":"2024-09-13T16:48:42.716932+0000","last_clean_scrub_stamp":"2024-09-13T16:48:42.716932+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:32.131509+0000","scrub_duration":0.0014958700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"22'183","reported_seq":209,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:35.636447+0000","last_change":"2024-09-13T16:48:35.636447+0000","last_active":"2024-09-13T16:48:35.636447+0000","last_peered":"2024-09-13T16:48:35.636447+0000","last_clean":"2024-09-13T16:48:35.636447+0000","last_became_active":"2024-09-13T16:43:54.986161+0000","last_became_peered":"2024-09-13T16:43:54.986161+0000","last_unstale":"2024-09-13T16:48:35.636447+0000","last_undegraded":"2024-09-13T16:48:35.636447+0000","last_fullsized":"2024-09-13T16:48:35.636447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'183","last_scrub_stamp":"2024-09-13T16:48:35.636320+0000","last_deep_scrub":"22'183","last_deep_scrub_stamp":"2024-09-13T16:48:35.636320+0000","last_clean_scrub_stamp":"2024-09-13T16:48:35.636320+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:19.361757+0000","scrub_duration":0.00160545,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"22'170","reported_seq":196,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:57.031257+0000","last_change":"2024-09-13T16:48:57.031362+0000","last_active":"2024-09-13T16:48:57.031257+0000","last_peered":"2024-09-13T16:48:57.031257+0000","last_clean":"2024-09-13T16:48:57.031257+0000","last_became_active":"2024-09-13T16:43:54.997162+0000","last_became_peered":"2024-09-13T16:43:54.997162+0000","last_unstale":"2024-09-13T16:48:57.031257+0000","last_undegraded":"2024-09-13T16:48:57.031257+0000","last_fullsized":"2024-09-13T16:48:57.031257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'170","last_scrub_stamp":"2024-09-13T16:48:57.031237+0000","last_deep_scrub":"22'170","last_deep_scrub_stamp":"2024-09-13T16:48:57.031237+0000","last_clean_scrub_stamp":"2024-09-13T16:48:57.031237+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:09.473170+0000","scrub_duration":0.0024071370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"22'180","reported_seq":206,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:38.508357+0000","last_change":"2024-09-13T16:48:38.508357+0000","last_active":"2024-09-13T16:48:38.508357+0000","last_peered":"2024-09-13T16:48:38.508357+0000","last_clean":"2024-09-13T16:48:38.508357+0000","last_became_active":"2024-09-13T16:43:54.994891+0000","last_became_peered":"2024-09-13T16:43:54.994891+0000","last_unstale":"2024-09-13T16:48:38.508357+0000","last_undegraded":"2024-09-13T16:48:38.508357+0000","last_fullsized":"2024-09-13T16:48:38.508357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'180","last_scrub_stamp":"2024-09-13T16:48:38.508228+0000","last_deep_scrub":"22'180","last_deep_scrub_stamp":"2024-09-13T16:48:38.508228+0000","last_clean_scrub_stamp":"2024-09-13T16:48:38.508228+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:27.922805+0000","scrub_duration":0.0019548199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"22'192","reported_seq":218,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:46.055489+0000","last_change":"2024-09-13T16:48:46.055589+0000","last_active":"2024-09-13T16:48:46.055489+0000","last_peered":"2024-09-13T16:48:46.055489+0000","last_clean":"2024-09-13T16:48:46.055489+0000","last_became_active":"2024-09-13T16:43:54.981191+0000","last_became_peered":"2024-09-13T16:43:54.981191+0000","last_unstale":"2024-09-13T16:48:46.055489+0000","last_undegraded":"2024-09-13T16:48:46.055489+0000","last_fullsized":"2024-09-13T16:48:46.055489+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'192","last_scrub_stamp":"2024-09-13T16:48:46.055465+0000","last_deep_scrub":"22'192","last_deep_scrub_stamp":"2024-09-13T16:48:46.055465+0000","last_clean_scrub_stamp":"2024-09-13T16:48:46.055465+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:02:47.540947+0000","scrub_duration":0.0026106329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"22'160","reported_seq":186,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:59.611478+0000","last_change":"2024-09-13T16:48:59.611575+0000","last_active":"2024-09-13T16:48:59.611478+0000","last_peered":"2024-09-13T16:48:59.611478+0000","last_clean":"2024-09-13T16:48:59.611478+0000","last_became_active":"2024-09-13T16:43:54.993915+0000","last_became_peered":"2024-09-13T16:43:54.993915+0000","last_unstale":"2024-09-13T16:48:59.611478+0000","last_undegraded":"2024-09-13T16:48:59.611478+0000","last_fullsized":"2024-09-13T16:48:59.611478+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'160","last_scrub_stamp":"2024-09-13T16:48:59.611456+0000","last_deep_scrub":"22'160","last_deep_scrub_stamp":"2024-09-13T16:48:59.611456+0000","last_clean_scrub_stamp":"2024-09-13T16:48:59.611456+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:56:45.404068+0000","scrub_duration":0.0023453189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"22'175","reported_seq":201,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:45.654046+0000","last_change":"2024-09-13T16:48:45.654180+0000","last_active":"2024-09-13T16:48:45.654046+0000","last_peered":"2024-09-13T16:48:45.654046+0000","last_clean":"2024-09-13T16:48:45.654046+0000","last_became_active":"2024-09-13T16:43:54.987048+0000","last_became_peered":"2024-09-13T16:43:54.987048+0000","last_unstale":"2024-09-13T16:48:45.654046+0000","last_undegraded":"2024-09-13T16:48:45.654046+0000","last_fullsized":"2024-09-13T16:48:45.654046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'175","last_scrub_stamp":"2024-09-13T16:48:45.654021+0000","last_deep_scrub":"22'175","last_deep_scrub_stamp":"2024-09-13T16:48:45.654021+0000","last_clean_scrub_stamp":"2024-09-13T16:48:45.654021+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:03.050434+0000","scrub_duration":0.001640159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"22'142","reported_seq":168,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:41.453063+0000","last_change":"2024-09-13T16:48:41.453200+0000","last_active":"2024-09-13T16:48:41.453063+0000","last_peered":"2024-09-13T16:48:41.453063+0000","last_clean":"2024-09-13T16:48:41.453063+0000","last_became_active":"2024-09-13T16:43:54.985378+0000","last_became_peered":"2024-09-13T16:43:54.985378+0000","last_unstale":"2024-09-13T16:48:41.453063+0000","last_undegraded":"2024-09-13T16:48:41.453063+0000","last_fullsized":"2024-09-13T16:48:41.453063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'142","last_scrub_stamp":"2024-09-13T16:48:41.453040+0000","last_deep_scrub":"22'142","last_deep_scrub_stamp":"2024-09-13T16:48:41.453040+0000","last_clean_scrub_stamp":"2024-09-13T16:48:41.453040+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:30.867617+0000","scrub_duration":0.0021012510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"22'157","reported_seq":183,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:34.019076+0000","last_change":"2024-09-13T16:48:34.019175+0000","last_active":"2024-09-13T16:48:34.019076+0000","last_peered":"2024-09-13T16:48:34.019076+0000","last_clean":"2024-09-13T16:48:34.019076+0000","last_became_active":"2024-09-13T16:43:54.996523+0000","last_became_peered":"2024-09-13T16:43:54.996523+0000","last_unstale":"2024-09-13T16:48:34.019076+0000","last_undegraded":"2024-09-13T16:48:34.019076+0000","last_fullsized":"2024-09-13T16:48:34.019076+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'157","last_scrub_stamp":"2024-09-13T16:48:34.019052+0000","last_deep_scrub":"22'157","last_deep_scrub_stamp":"2024-09-13T16:48:34.019052+0000","last_clean_scrub_stamp":"2024-09-13T16:48:34.019052+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:10.737127+0000","scrub_duration":0.0021886039999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"22'167","reported_seq":193,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:54.590636+0000","last_change":"2024-09-13T16:48:54.590740+0000","last_active":"2024-09-13T16:48:54.590636+0000","last_peered":"2024-09-13T16:48:54.590636+0000","last_clean":"2024-09-13T16:48:54.590636+0000","last_became_active":"2024-09-13T16:43:54.984595+0000","last_became_peered":"2024-09-13T16:43:54.984595+0000","last_unstale":"2024-09-13T16:48:54.590636+0000","last_undegraded":"2024-09-13T16:48:54.590636+0000","last_fullsized":"2024-09-13T16:48:54.590636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'167","last_scrub_stamp":"2024-09-13T16:48:54.590613+0000","last_deep_scrub":"22'167","last_deep_scrub_stamp":"2024-09-13T16:48:54.590613+0000","last_clean_scrub_stamp":"2024-09-13T16:48:54.590613+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:02:56.076095+0000","scrub_duration":0.001613223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"22'167","reported_seq":193,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:48.554656+0000","last_change":"2024-09-13T16:48:48.554774+0000","last_active":"2024-09-13T16:48:48.554656+0000","last_peered":"2024-09-13T16:48:48.554656+0000","last_clean":"2024-09-13T16:48:48.554656+0000","last_became_active":"2024-09-13T16:43:54.996275+0000","last_became_peered":"2024-09-13T16:43:54.996275+0000","last_unstale":"2024-09-13T16:48:48.554656+0000","last_undegraded":"2024-09-13T16:48:48.554656+0000","last_fullsized":"2024-09-13T16:48:48.554656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'167","last_scrub_stamp":"2024-09-13T16:48:48.554633+0000","last_deep_scrub":"22'167","last_deep_scrub_stamp":"2024-09-13T16:48:48.554633+0000","last_clean_scrub_stamp":"2024-09-13T16:48:48.554633+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:08:39.168750+0000","scrub_duration":0.0023296100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"22'199","reported_seq":225,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:49:02.612473+0000","last_change":"2024-09-13T16:49:02.612573+0000","last_active":"2024-09-13T16:49:02.612473+0000","last_peered":"2024-09-13T16:49:02.612473+0000","last_clean":"2024-09-13T16:49:02.612473+0000","last_became_active":"2024-09-13T16:43:54.985130+0000","last_became_peered":"2024-09-13T16:43:54.985130+0000","last_unstale":"2024-09-13T16:49:02.612473+0000","last_undegraded":"2024-09-13T16:49:02.612473+0000","last_fullsized":"2024-09-13T16:49:02.612473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'199","last_scrub_stamp":"2024-09-13T16:49:02.612449+0000","last_deep_scrub":"22'199","last_deep_scrub_stamp":"2024-09-13T16:49:02.612449+0000","last_clean_scrub_stamp":"2024-09-13T16:49:02.612449+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:40.415636+0000","scrub_duration":0.0015657539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"22'158","reported_seq":184,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:55.038257+0000","last_change":"2024-09-13T16:48:55.038379+0000","last_active":"2024-09-13T16:48:55.038257+0000","last_peered":"2024-09-13T16:48:55.038257+0000","last_clean":"2024-09-13T16:48:55.038257+0000","last_became_active":"2024-09-13T16:43:54.991554+0000","last_became_peered":"2024-09-13T16:43:54.991554+0000","last_unstale":"2024-09-13T16:48:55.038257+0000","last_undegraded":"2024-09-13T16:48:55.038257+0000","last_fullsized":"2024-09-13T16:48:55.038257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'158","last_scrub_stamp":"2024-09-13T16:48:55.038236+0000","last_deep_scrub":"22'158","last_deep_scrub_stamp":"2024-09-13T16:48:55.038236+0000","last_clean_scrub_stamp":"2024-09-13T16:48:55.038236+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:54.428660+0000","scrub_duration":0.0021259730000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"22'162","reported_seq":185,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:46.528414+0000","last_change":"2024-09-13T16:48:46.528520+0000","last_active":"2024-09-13T16:48:46.528414+0000","last_peered":"2024-09-13T16:48:46.528414+0000","last_clean":"2024-09-13T16:48:46.528414+0000","last_became_active":"2024-09-13T16:44:59.220762+0000","last_became_peered":"2024-09-13T16:44:59.220762+0000","last_unstale":"2024-09-13T16:48:46.528414+0000","last_undegraded":"2024-09-13T16:48:46.528414+0000","last_fullsized":"2024-09-13T16:48:46.528414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'162","last_scrub_stamp":"2024-09-13T16:48:46.528394+0000","last_deep_scrub":"22'162","last_deep_scrub_stamp":"2024-09-13T16:48:46.528394+0000","last_clean_scrub_stamp":"2024-09-13T16:48:46.528394+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:36:53.838527+0000","scrub_duration":0.0017451000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"22'170","reported_seq":196,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:46.934429+0000","last_change":"2024-09-13T16:48:46.934624+0000","last_active":"2024-09-13T16:48:46.934429+0000","last_peered":"2024-09-13T16:48:46.934429+0000","last_clean":"2024-09-13T16:48:46.934429+0000","last_became_active":"2024-09-13T16:43:54.991127+0000","last_became_peered":"2024-09-13T16:43:54.991127+0000","last_unstale":"2024-09-13T16:48:46.934429+0000","last_undegraded":"2024-09-13T16:48:46.934429+0000","last_fullsized":"2024-09-13T16:48:46.934429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'170","last_scrub_stamp":"2024-09-13T16:48:46.934406+0000","last_deep_scrub":"22'170","last_deep_scrub_stamp":"2024-09-13T16:48:46.934406+0000","last_clean_scrub_stamp":"2024-09-13T16:48:46.934406+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:08:37.548523+0000","scrub_duration":0.0024823760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"22'180","reported_seq":206,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:47.738029+0000","last_change":"2024-09-13T16:48:47.738136+0000","last_active":"2024-09-13T16:48:47.738029+0000","last_peered":"2024-09-13T16:48:47.738029+0000","last_clean":"2024-09-13T16:48:47.738029+0000","last_became_active":"2024-09-13T16:43:54.985299+0000","last_became_peered":"2024-09-13T16:43:54.985299+0000","last_unstale":"2024-09-13T16:48:47.738029+0000","last_undegraded":"2024-09-13T16:48:47.738029+0000","last_fullsized":"2024-09-13T16:48:47.738029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'180","last_scrub_stamp":"2024-09-13T16:48:47.738005+0000","last_deep_scrub":"22'180","last_deep_scrub_stamp":"2024-09-13T16:48:47.738005+0000","last_clean_scrub_stamp":"2024-09-13T16:48:47.738005+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:36:55.048139+0000","scrub_duration":0.001558622,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"22'147","reported_seq":173,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:51.046402+0000","last_change":"2024-09-13T16:48:51.046479+0000","last_active":"2024-09-13T16:48:51.046402+0000","last_peered":"2024-09-13T16:48:51.046402+0000","last_clean":"2024-09-13T16:48:51.046402+0000","last_became_active":"2024-09-13T16:43:54.993009+0000","last_became_peered":"2024-09-13T16:43:54.993009+0000","last_unstale":"2024-09-13T16:48:51.046402+0000","last_undegraded":"2024-09-13T16:48:51.046402+0000","last_fullsized":"2024-09-13T16:48:51.046402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'147","last_scrub_stamp":"2024-09-13T16:48:51.046384+0000","last_deep_scrub":"22'147","last_deep_scrub_stamp":"2024-09-13T16:48:51.046384+0000","last_clean_scrub_stamp":"2024-09-13T16:48:51.046384+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:50.436807+0000","scrub_duration":0.001304983,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"22'188","reported_seq":214,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:36.902401+0000","last_change":"2024-09-13T16:48:36.902509+0000","last_active":"2024-09-13T16:48:36.902401+0000","last_peered":"2024-09-13T16:48:36.902401+0000","last_clean":"2024-09-13T16:48:36.902401+0000","last_became_active":"2024-09-13T16:43:54.995774+0000","last_became_peered":"2024-09-13T16:43:54.995774+0000","last_unstale":"2024-09-13T16:48:36.902401+0000","last_undegraded":"2024-09-13T16:48:36.902401+0000","last_fullsized":"2024-09-13T16:48:36.902401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'188","last_scrub_stamp":"2024-09-13T16:48:36.902377+0000","last_deep_scrub":"22'188","last_deep_scrub_stamp":"2024-09-13T16:48:36.902377+0000","last_clean_scrub_stamp":"2024-09-13T16:48:36.902377+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:26.316954+0000","scrub_duration":0.0025868390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"22'172","reported_seq":198,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:45.453576+0000","last_change":"2024-09-13T16:48:45.453673+0000","last_active":"2024-09-13T16:48:45.453576+0000","last_peered":"2024-09-13T16:48:45.453576+0000","last_clean":"2024-09-13T16:48:45.453576+0000","last_became_active":"2024-09-13T16:43:55.001713+0000","last_became_peered":"2024-09-13T16:43:55.001713+0000","last_unstale":"2024-09-13T16:48:45.453576+0000","last_undegraded":"2024-09-13T16:48:45.453576+0000","last_fullsized":"2024-09-13T16:48:45.453576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'172","last_scrub_stamp":"2024-09-13T16:48:45.453552+0000","last_deep_scrub":"22'172","last_deep_scrub_stamp":"2024-09-13T16:48:45.453552+0000","last_clean_scrub_stamp":"2024-09-13T16:48:45.453552+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:24.544274+0000","scrub_duration":0.0017292360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"22'156","reported_seq":182,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:49:01.630196+0000","last_change":"2024-09-13T16:49:01.630304+0000","last_active":"2024-09-13T16:49:01.630196+0000","last_peered":"2024-09-13T16:49:01.630196+0000","last_clean":"2024-09-13T16:49:01.630196+0000","last_became_active":"2024-09-13T16:43:54.992508+0000","last_became_peered":"2024-09-13T16:43:54.992508+0000","last_unstale":"2024-09-13T16:49:01.630196+0000","last_undegraded":"2024-09-13T16:49:01.630196+0000","last_fullsized":"2024-09-13T16:49:01.630196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'156","last_scrub_stamp":"2024-09-13T16:49:01.630156+0000","last_deep_scrub":"22'156","last_deep_scrub_stamp":"2024-09-13T16:49:01.630156+0000","last_clean_scrub_stamp":"2024-09-13T16:49:01.630156+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:07.952642+0000","scrub_duration":0.002174886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"22'143","reported_seq":169,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:40.496590+0000","last_change":"2024-09-13T16:48:40.496687+0000","last_active":"2024-09-13T16:48:40.496590+0000","last_peered":"2024-09-13T16:48:40.496590+0000","last_clean":"2024-09-13T16:48:40.496590+0000","last_became_active":"2024-09-13T16:43:54.986277+0000","last_became_peered":"2024-09-13T16:43:54.986277+0000","last_unstale":"2024-09-13T16:48:40.496590+0000","last_undegraded":"2024-09-13T16:48:40.496590+0000","last_fullsized":"2024-09-13T16:48:40.496590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'143","last_scrub_stamp":"2024-09-13T16:48:40.496565+0000","last_deep_scrub":"22'143","last_deep_scrub_stamp":"2024-09-13T16:48:40.496565+0000","last_clean_scrub_stamp":"2024-09-13T16:48:40.496565+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:29.395789+0000","scrub_duration":0.0022908500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"22'184","reported_seq":210,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:55.582331+0000","last_change":"2024-09-13T16:48:55.582455+0000","last_active":"2024-09-13T16:48:55.582331+0000","last_peered":"2024-09-13T16:48:55.582331+0000","last_clean":"2024-09-13T16:48:55.582331+0000","last_became_active":"2024-09-13T16:43:54.999778+0000","last_became_peered":"2024-09-13T16:43:54.999778+0000","last_unstale":"2024-09-13T16:48:55.582331+0000","last_undegraded":"2024-09-13T16:48:55.582331+0000","last_fullsized":"2024-09-13T16:48:55.582331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'184","last_scrub_stamp":"2024-09-13T16:48:55.582308+0000","last_deep_scrub":"22'184","last_deep_scrub_stamp":"2024-09-13T16:48:55.582308+0000","last_clean_scrub_stamp":"2024-09-13T16:48:55.582308+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:23.331793+0000","scrub_duration":0.0020665079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"22'219","reported_seq":245,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:32.570000+0000","last_change":"2024-09-13T16:48:32.570000+0000","last_active":"2024-09-13T16:48:32.570000+0000","last_peered":"2024-09-13T16:48:32.570000+0000","last_clean":"2024-09-13T16:48:32.570000+0000","last_became_active":"2024-09-13T16:43:54.988131+0000","last_became_peered":"2024-09-13T16:43:54.988131+0000","last_unstale":"2024-09-13T16:48:32.570000+0000","last_undegraded":"2024-09-13T16:48:32.570000+0000","last_fullsized":"2024-09-13T16:48:32.570000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'219","last_scrub_stamp":"2024-09-13T16:48:32.569873+0000","last_deep_scrub":"22'219","last_deep_scrub_stamp":"2024-09-13T16:48:32.569873+0000","last_clean_scrub_stamp":"2024-09-13T16:48:32.569873+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:13.799800+0000","scrub_duration":0.0023362790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'7","reported_seq":34,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:34.875982+0000","last_change":"2024-09-13T16:48:34.875982+0000","last_active":"2024-09-13T16:48:34.875982+0000","last_peered":"2024-09-13T16:48:34.875982+0000","last_clean":"2024-09-13T16:48:34.875982+0000","last_became_active":"2024-09-13T16:43:54.554697+0000","last_became_peered":"2024-09-13T16:43:54.554697+0000","last_unstale":"2024-09-13T16:48:34.875982+0000","last_undegraded":"2024-09-13T16:48:34.875982+0000","last_fullsized":"2024-09-13T16:48:34.875982+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'7","last_scrub_stamp":"2024-09-13T16:48:34.875828+0000","last_deep_scrub":"22'7","last_deep_scrub_stamp":"2024-09-13T16:48:34.875828+0000","last_clean_scrub_stamp":"2024-09-13T16:48:34.875828+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:12.773199+0000","scrub_duration":0.0042325979999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"22'173","reported_seq":199,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:43.449335+0000","last_change":"2024-09-13T16:48:43.449335+0000","last_active":"2024-09-13T16:48:43.449335+0000","last_peered":"2024-09-13T16:48:43.449335+0000","last_clean":"2024-09-13T16:48:43.449335+0000","last_became_active":"2024-09-13T16:43:54.998399+0000","last_became_peered":"2024-09-13T16:43:54.998399+0000","last_unstale":"2024-09-13T16:48:43.449335+0000","last_undegraded":"2024-09-13T16:48:43.449335+0000","last_fullsized":"2024-09-13T16:48:43.449335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'173","last_scrub_stamp":"2024-09-13T16:48:43.449202+0000","last_deep_scrub":"22'173","last_deep_scrub_stamp":"2024-09-13T16:48:43.449202+0000","last_clean_scrub_stamp":"2024-09-13T16:48:43.449202+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:41.461235+0000","scrub_duration":0.0019285179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'10","reported_seq":37,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:31.822764+0000","last_change":"2024-09-13T16:48:31.822764+0000","last_active":"2024-09-13T16:48:31.822764+0000","last_peered":"2024-09-13T16:48:31.822764+0000","last_clean":"2024-09-13T16:48:31.822764+0000","last_became_active":"2024-09-13T16:43:54.552977+0000","last_became_peered":"2024-09-13T16:43:54.552977+0000","last_unstale":"2024-09-13T16:48:31.822764+0000","last_undegraded":"2024-09-13T16:48:31.822764+0000","last_fullsized":"2024-09-13T16:48:31.822764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'10","last_scrub_stamp":"2024-09-13T16:48:31.822629+0000","last_deep_scrub":"22'10","last_deep_scrub_stamp":"2024-09-13T16:48:31.822629+0000","last_clean_scrub_stamp":"2024-09-13T16:48:31.822629+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:13.052556+0000","scrub_duration":0.0055532949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"22'5","reported_seq":32,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:28.811089+0000","last_change":"2024-09-13T16:48:28.811089+0000","last_active":"2024-09-13T16:48:28.811089+0000","last_peered":"2024-09-13T16:48:28.811089+0000","last_clean":"2024-09-13T16:48:28.811089+0000","last_became_active":"2024-09-13T16:43:54.557722+0000","last_became_peered":"2024-09-13T16:43:54.557722+0000","last_unstale":"2024-09-13T16:48:28.811089+0000","last_undegraded":"2024-09-13T16:48:28.811089+0000","last_fullsized":"2024-09-13T16:48:28.811089+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'5","last_scrub_stamp":"2024-09-13T16:48:28.810988+0000","last_deep_scrub":"22'5","last_deep_scrub_stamp":"2024-09-13T16:48:28.810988+0000","last_clean_scrub_stamp":"2024-09-13T16:48:28.810988+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:12.536424+0000","scrub_duration":0.0035919440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"22'159","reported_seq":185,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:49.952391+0000","last_change":"2024-09-13T16:48:49.952504+0000","last_active":"2024-09-13T16:48:49.952391+0000","last_peered":"2024-09-13T16:48:49.952391+0000","last_clean":"2024-09-13T16:48:49.952391+0000","last_became_active":"2024-09-13T16:43:55.001512+0000","last_became_peered":"2024-09-13T16:43:55.001512+0000","last_unstale":"2024-09-13T16:48:49.952391+0000","last_undegraded":"2024-09-13T16:48:49.952391+0000","last_fullsized":"2024-09-13T16:48:49.952391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'159","last_scrub_stamp":"2024-09-13T16:48:49.952367+0000","last_deep_scrub":"22'159","last_deep_scrub_stamp":"2024-09-13T16:48:49.952367+0000","last_clean_scrub_stamp":"2024-09-13T16:48:49.952367+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:48:04.317934+0000","scrub_duration":0.0017871230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"22'4","reported_seq":31,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:31.538047+0000","last_change":"2024-09-13T16:48:31.538047+0000","last_active":"2024-09-13T16:48:31.538047+0000","last_peered":"2024-09-13T16:48:31.538047+0000","last_clean":"2024-09-13T16:48:31.538047+0000","last_became_active":"2024-09-13T16:43:54.550154+0000","last_became_peered":"2024-09-13T16:43:54.550154+0000","last_unstale":"2024-09-13T16:48:31.538047+0000","last_undegraded":"2024-09-13T16:48:31.538047+0000","last_fullsized":"2024-09-13T16:48:31.538047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'4","last_scrub_stamp":"2024-09-13T16:48:31.537892+0000","last_deep_scrub":"22'4","last_deep_scrub_stamp":"2024-09-13T16:48:31.537892+0000","last_clean_scrub_stamp":"2024-09-13T16:48:31.537892+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:09.435264+0000","scrub_duration":0.0031102679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"22'178","reported_seq":204,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:48.011383+0000","last_change":"2024-09-13T16:48:48.011383+0000","last_active":"2024-09-13T16:48:48.011383+0000","last_peered":"2024-09-13T16:48:48.011383+0000","last_clean":"2024-09-13T16:48:48.011383+0000","last_became_active":"2024-09-13T16:43:54.993036+0000","last_became_peered":"2024-09-13T16:43:54.993036+0000","last_unstale":"2024-09-13T16:48:48.011383+0000","last_undegraded":"2024-09-13T16:48:48.011383+0000","last_fullsized":"2024-09-13T16:48:48.011383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'178","last_scrub_stamp":"2024-09-13T16:48:48.011264+0000","last_deep_scrub":"22'178","last_deep_scrub_stamp":"2024-09-13T16:48:48.011264+0000","last_clean_scrub_stamp":"2024-09-13T16:48:48.011264+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:19.491965+0000","scrub_duration":0.001649017,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'8","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:40.755255+0000","last_change":"2024-09-13T16:48:40.755255+0000","last_active":"2024-09-13T16:48:40.755255+0000","last_peered":"2024-09-13T16:48:40.755255+0000","last_clean":"2024-09-13T16:48:40.755255+0000","last_became_active":"2024-09-13T16:43:54.555267+0000","last_became_peered":"2024-09-13T16:43:54.555267+0000","last_unstale":"2024-09-13T16:48:40.755255+0000","last_undegraded":"2024-09-13T16:48:40.755255+0000","last_fullsized":"2024-09-13T16:48:40.755255+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'8","last_scrub_stamp":"2024-09-13T16:48:40.755129+0000","last_deep_scrub":"22'8","last_deep_scrub_stamp":"2024-09-13T16:48:40.755129+0000","last_clean_scrub_stamp":"2024-09-13T16:48:40.755129+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:18.652500+0000","scrub_duration":0.0039776960000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"22'153","reported_seq":179,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:43.627266+0000","last_change":"2024-09-13T16:48:43.627266+0000","last_active":"2024-09-13T16:48:43.627266+0000","last_peered":"2024-09-13T16:48:43.627266+0000","last_clean":"2024-09-13T16:48:43.627266+0000","last_became_active":"2024-09-13T16:43:54.981429+0000","last_became_peered":"2024-09-13T16:43:54.981429+0000","last_unstale":"2024-09-13T16:48:43.627266+0000","last_undegraded":"2024-09-13T16:48:43.627266+0000","last_fullsized":"2024-09-13T16:48:43.627266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'153","last_scrub_stamp":"2024-09-13T16:48:43.627115+0000","last_deep_scrub":"22'153","last_deep_scrub_stamp":"2024-09-13T16:48:43.627115+0000","last_clean_scrub_stamp":"2024-09-13T16:48:43.627115+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:08:34.241232+0000","scrub_duration":0.001694222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'6","reported_seq":33,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:33.542809+0000","last_change":"2024-09-13T16:48:33.542809+0000","last_active":"2024-09-13T16:48:33.542809+0000","last_peered":"2024-09-13T16:48:33.542809+0000","last_clean":"2024-09-13T16:48:33.542809+0000","last_became_active":"2024-09-13T16:43:54.554307+0000","last_became_peered":"2024-09-13T16:43:54.554307+0000","last_unstale":"2024-09-13T16:48:33.542809+0000","last_undegraded":"2024-09-13T16:48:33.542809+0000","last_fullsized":"2024-09-13T16:48:33.542809+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-09-13T16:48:33.542675+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-09-13T16:48:33.542675+0000","last_clean_scrub_stamp":"2024-09-13T16:48:33.542675+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:22.957253+0000","scrub_duration":0.003084865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"22'204","reported_seq":230,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:50.579535+0000","last_change":"2024-09-13T16:48:50.579612+0000","last_active":"2024-09-13T16:48:50.579535+0000","last_peered":"2024-09-13T16:48:50.579535+0000","last_clean":"2024-09-13T16:48:50.579535+0000","last_became_active":"2024-09-13T16:43:54.993226+0000","last_became_peered":"2024-09-13T16:43:54.993226+0000","last_unstale":"2024-09-13T16:48:50.579535+0000","last_undegraded":"2024-09-13T16:48:50.579535+0000","last_fullsized":"2024-09-13T16:48:50.579535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'204","last_scrub_stamp":"2024-09-13T16:48:50.579515+0000","last_deep_scrub":"22'204","last_deep_scrub_stamp":"2024-09-13T16:48:50.579515+0000","last_clean_scrub_stamp":"2024-09-13T16:48:50.579515+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:07.975927+0000","scrub_duration":0.0020192729999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'15","reported_seq":42,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:43.064614+0000","last_change":"2024-09-13T16:48:43.064614+0000","last_active":"2024-09-13T16:48:43.064614+0000","last_peered":"2024-09-13T16:48:43.064614+0000","last_clean":"2024-09-13T16:48:43.064614+0000","last_became_active":"2024-09-13T16:43:54.560480+0000","last_became_peered":"2024-09-13T16:43:54.560480+0000","last_unstale":"2024-09-13T16:48:43.064614+0000","last_undegraded":"2024-09-13T16:48:43.064614+0000","last_fullsized":"2024-09-13T16:48:43.064614+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-09-13T16:48:43.064502+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-09-13T16:48:43.064502+0000","last_clean_scrub_stamp":"2024-09-13T16:48:43.064502+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:22.155224+0000","scrub_duration":0.0045066539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"22'211","reported_seq":237,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:29.633706+0000","last_change":"2024-09-13T16:48:29.633706+0000","last_active":"2024-09-13T16:48:29.633706+0000","last_peered":"2024-09-13T16:48:29.633706+0000","last_clean":"2024-09-13T16:48:29.633706+0000","last_became_active":"2024-09-13T16:43:54.986083+0000","last_became_peered":"2024-09-13T16:43:54.986083+0000","last_unstale":"2024-09-13T16:48:29.633706+0000","last_undegraded":"2024-09-13T16:48:29.633706+0000","last_fullsized":"2024-09-13T16:48:29.633706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'211","last_scrub_stamp":"2024-09-13T16:48:29.633578+0000","last_deep_scrub":"22'211","last_deep_scrub_stamp":"2024-09-13T16:48:29.633578+0000","last_clean_scrub_stamp":"2024-09-13T16:48:29.633578+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:02.991594+0000","scrub_duration":0.001550811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'171","reported_seq":198,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:37.744830+0000","last_change":"2024-09-13T16:48:37.744926+0000","last_active":"2024-09-13T16:48:37.744830+0000","last_peered":"2024-09-13T16:48:37.744830+0000","last_clean":"2024-09-13T16:48:37.744830+0000","last_became_active":"2024-09-13T16:43:54.565701+0000","last_became_peered":"2024-09-13T16:43:54.565701+0000","last_unstale":"2024-09-13T16:48:37.744830+0000","last_undegraded":"2024-09-13T16:48:37.744830+0000","last_fullsized":"2024-09-13T16:48:37.744830+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'171","last_scrub_stamp":"2024-09-13T16:48:37.744806+0000","last_deep_scrub":"22'171","last_deep_scrub_stamp":"2024-09-13T16:48:37.744806+0000","last_clean_scrub_stamp":"2024-09-13T16:48:37.744806+0000","objects_scrubbed":6,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:10:01.111758+0000","scrub_duration":0.0062340260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364698,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"22'241","reported_seq":267,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:49:00.976438+0000","last_change":"2024-09-13T16:49:00.976538+0000","last_active":"2024-09-13T16:49:00.976438+0000","last_peered":"2024-09-13T16:49:00.976438+0000","last_clean":"2024-09-13T16:49:00.976438+0000","last_became_active":"2024-09-13T16:43:54.988779+0000","last_became_peered":"2024-09-13T16:43:54.988779+0000","last_unstale":"2024-09-13T16:49:00.976438+0000","last_undegraded":"2024-09-13T16:49:00.976438+0000","last_fullsized":"2024-09-13T16:49:00.976438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'241","last_scrub_stamp":"2024-09-13T16:49:00.976413+0000","last_deep_scrub":"22'241","last_deep_scrub_stamp":"2024-09-13T16:49:00.976413+0000","last_clean_scrub_stamp":"2024-09-13T16:49:00.976413+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:56:46.769026+0000","scrub_duration":0.002323393,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":213,"num_write_kb":581632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'15","reported_seq":42,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:28.421871+0000","last_change":"2024-09-13T16:48:28.421871+0000","last_active":"2024-09-13T16:48:28.421871+0000","last_peered":"2024-09-13T16:48:28.421871+0000","last_clean":"2024-09-13T16:48:28.421871+0000","last_became_active":"2024-09-13T16:43:54.561077+0000","last_became_peered":"2024-09-13T16:43:54.561077+0000","last_unstale":"2024-09-13T16:48:28.421871+0000","last_undegraded":"2024-09-13T16:48:28.421871+0000","last_fullsized":"2024-09-13T16:48:28.421871+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-09-13T16:48:28.421778+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-09-13T16:48:28.421778+0000","last_clean_scrub_stamp":"2024-09-13T16:48:28.421778+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:09.651705+0000","scrub_duration":0.0051464910000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"22'162","reported_seq":188,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:31.409973+0000","last_change":"2024-09-13T16:48:31.409973+0000","last_active":"2024-09-13T16:48:31.409973+0000","last_peered":"2024-09-13T16:48:31.409973+0000","last_clean":"2024-09-13T16:48:31.409973+0000","last_became_active":"2024-09-13T16:43:54.999283+0000","last_became_peered":"2024-09-13T16:43:54.999283+0000","last_unstale":"2024-09-13T16:48:31.409973+0000","last_undegraded":"2024-09-13T16:48:31.409973+0000","last_fullsized":"2024-09-13T16:48:31.409973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'162","last_scrub_stamp":"2024-09-13T16:48:31.409830+0000","last_deep_scrub":"22'162","last_deep_scrub_stamp":"2024-09-13T16:48:31.409830+0000","last_clean_scrub_stamp":"2024-09-13T16:48:31.409830+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:15.135267+0000","scrub_duration":0.002545552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"22'8","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:38.572732+0000","last_change":"2024-09-13T16:48:38.572847+0000","last_active":"2024-09-13T16:48:38.572732+0000","last_peered":"2024-09-13T16:48:38.572732+0000","last_clean":"2024-09-13T16:48:38.572732+0000","last_became_active":"2024-09-13T16:43:54.542548+0000","last_became_peered":"2024-09-13T16:43:54.542548+0000","last_unstale":"2024-09-13T16:48:38.572732+0000","last_undegraded":"2024-09-13T16:48:38.572732+0000","last_fullsized":"2024-09-13T16:48:38.572732+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'8","last_scrub_stamp":"2024-09-13T16:48:38.572702+0000","last_deep_scrub":"22'8","last_deep_scrub_stamp":"2024-09-13T16:48:38.572702+0000","last_clean_scrub_stamp":"2024-09-13T16:48:38.572702+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:36.584734+0000","scrub_duration":0.0044139139999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"22'195","reported_seq":221,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:55.950658+0000","last_change":"2024-09-13T16:48:55.950766+0000","last_active":"2024-09-13T16:48:55.950658+0000","last_peered":"2024-09-13T16:48:55.950658+0000","last_clean":"2024-09-13T16:48:55.950658+0000","last_became_active":"2024-09-13T16:43:54.988716+0000","last_became_peered":"2024-09-13T16:43:54.988716+0000","last_unstale":"2024-09-13T16:48:55.950658+0000","last_undegraded":"2024-09-13T16:48:55.950658+0000","last_fullsized":"2024-09-13T16:48:55.950658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'195","last_scrub_stamp":"2024-09-13T16:48:55.950628+0000","last_deep_scrub":"22'195","last_deep_scrub_stamp":"2024-09-13T16:48:55.950628+0000","last_clean_scrub_stamp":"2024-09-13T16:48:55.950628+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:33.289014+0000","scrub_duration":0.001441037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'9","reported_seq":36,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:35.743470+0000","last_change":"2024-09-13T16:48:35.743577+0000","last_active":"2024-09-13T16:48:35.743470+0000","last_peered":"2024-09-13T16:48:35.743470+0000","last_clean":"2024-09-13T16:48:35.743470+0000","last_became_active":"2024-09-13T16:43:54.551293+0000","last_became_peered":"2024-09-13T16:43:54.551293+0000","last_unstale":"2024-09-13T16:48:35.743470+0000","last_undegraded":"2024-09-13T16:48:35.743470+0000","last_fullsized":"2024-09-13T16:48:35.743470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-09-13T16:48:35.743447+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-09-13T16:48:35.743447+0000","last_clean_scrub_stamp":"2024-09-13T16:48:35.743447+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:19.468883+0000","scrub_duration":0.0033744299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":940,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"22'169","reported_seq":195,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:41.738426+0000","last_change":"2024-09-13T16:48:41.738532+0000","last_active":"2024-09-13T16:48:41.738426+0000","last_peered":"2024-09-13T16:48:41.738426+0000","last_clean":"2024-09-13T16:48:41.738426+0000","last_became_active":"2024-09-13T16:43:54.994245+0000","last_became_peered":"2024-09-13T16:43:54.994245+0000","last_unstale":"2024-09-13T16:48:41.738426+0000","last_undegraded":"2024-09-13T16:48:41.738426+0000","last_fullsized":"2024-09-13T16:48:41.738426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'169","last_scrub_stamp":"2024-09-13T16:48:41.738403+0000","last_deep_scrub":"22'169","last_deep_scrub_stamp":"2024-09-13T16:48:41.738403+0000","last_clean_scrub_stamp":"2024-09-13T16:48:41.738403+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:30.637627+0000","scrub_duration":0.002340941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'26","reported_seq":53,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:34.429476+0000","last_change":"2024-09-13T16:48:34.429476+0000","last_active":"2024-09-13T16:48:34.429476+0000","last_peered":"2024-09-13T16:48:34.429476+0000","last_clean":"2024-09-13T16:48:34.429476+0000","last_became_active":"2024-09-13T16:43:54.561939+0000","last_became_peered":"2024-09-13T16:43:54.561939+0000","last_unstale":"2024-09-13T16:48:34.429476+0000","last_undegraded":"2024-09-13T16:48:34.429476+0000","last_fullsized":"2024-09-13T16:48:34.429476+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'26","last_scrub_stamp":"2024-09-13T16:48:34.429340+0000","last_deep_scrub":"22'26","last_deep_scrub_stamp":"2024-09-13T16:48:34.429340+0000","last_clean_scrub_stamp":"2024-09-13T16:48:34.429340+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:18.154777+0000","scrub_duration":0.006433701,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"22'175","reported_seq":201,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:48.910813+0000","last_change":"2024-09-13T16:48:48.910907+0000","last_active":"2024-09-13T16:48:48.910813+0000","last_peered":"2024-09-13T16:48:48.910813+0000","last_clean":"2024-09-13T16:48:48.910813+0000","last_became_active":"2024-09-13T16:43:54.988172+0000","last_became_peered":"2024-09-13T16:43:54.988172+0000","last_unstale":"2024-09-13T16:48:48.910813+0000","last_undegraded":"2024-09-13T16:48:48.910813+0000","last_fullsized":"2024-09-13T16:48:48.910813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'175","last_scrub_stamp":"2024-09-13T16:48:48.910789+0000","last_deep_scrub":"22'175","last_deep_scrub_stamp":"2024-09-13T16:48:48.910789+0000","last_clean_scrub_stamp":"2024-09-13T16:48:48.910789+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:16.660275+0000","scrub_duration":0.0015205030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'5","reported_seq":32,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:28.623118+0000","last_change":"2024-09-13T16:48:28.623229+0000","last_active":"2024-09-13T16:48:28.623118+0000","last_peered":"2024-09-13T16:48:28.623118+0000","last_clean":"2024-09-13T16:48:28.623118+0000","last_became_active":"2024-09-13T16:43:54.559111+0000","last_became_peered":"2024-09-13T16:43:54.559111+0000","last_unstale":"2024-09-13T16:48:28.623118+0000","last_undegraded":"2024-09-13T16:48:28.623118+0000","last_fullsized":"2024-09-13T16:48:28.623118+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'5","last_scrub_stamp":"2024-09-13T16:48:28.623093+0000","last_deep_scrub":"22'5","last_deep_scrub_stamp":"2024-09-13T16:48:28.623093+0000","last_clean_scrub_stamp":"2024-09-13T16:48:28.623093+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:43.512775+0000","scrub_duration":0.0030479550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'156","reported_seq":182,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:52.933867+0000","last_change":"2024-09-13T16:48:52.933962+0000","last_active":"2024-09-13T16:48:52.933867+0000","last_peered":"2024-09-13T16:48:52.933867+0000","last_clean":"2024-09-13T16:48:52.933867+0000","last_became_active":"2024-09-13T16:43:54.988059+0000","last_became_peered":"2024-09-13T16:43:54.988059+0000","last_unstale":"2024-09-13T16:48:52.933867+0000","last_undegraded":"2024-09-13T16:48:52.933867+0000","last_fullsized":"2024-09-13T16:48:52.933867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'156","last_scrub_stamp":"2024-09-13T16:48:52.933843+0000","last_deep_scrub":"22'156","last_deep_scrub_stamp":"2024-09-13T16:48:52.933843+0000","last_clean_scrub_stamp":"2024-09-13T16:48:52.933843+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:52.324266+0000","scrub_duration":0.001628792,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'7","reported_seq":34,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:39.488130+0000","last_change":"2024-09-13T16:48:39.488234+0000","last_active":"2024-09-13T16:48:39.488130+0000","last_peered":"2024-09-13T16:48:39.488130+0000","last_clean":"2024-09-13T16:48:39.488130+0000","last_became_active":"2024-09-13T16:43:54.565861+0000","last_became_peered":"2024-09-13T16:43:54.565861+0000","last_unstale":"2024-09-13T16:48:39.488130+0000","last_undegraded":"2024-09-13T16:48:39.488130+0000","last_fullsized":"2024-09-13T16:48:39.488130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'7","last_scrub_stamp":"2024-09-13T16:48:39.488109+0000","last_deep_scrub":"22'7","last_deep_scrub_stamp":"2024-09-13T16:48:39.488109+0000","last_clean_scrub_stamp":"2024-09-13T16:48:39.488109+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:17.385480+0000","scrub_duration":0.0035285960000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"22'136","reported_seq":159,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:51.551383+0000","last_change":"2024-09-13T16:48:51.551504+0000","last_active":"2024-09-13T16:48:51.551383+0000","last_peered":"2024-09-13T16:48:51.551383+0000","last_clean":"2024-09-13T16:48:51.551383+0000","last_became_active":"2024-09-13T16:44:58.956557+0000","last_became_peered":"2024-09-13T16:44:58.956557+0000","last_unstale":"2024-09-13T16:48:51.551383+0000","last_undegraded":"2024-09-13T16:48:51.551383+0000","last_fullsized":"2024-09-13T16:48:51.551383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'136","last_scrub_stamp":"2024-09-13T16:48:51.551365+0000","last_deep_scrub":"22'136","last_deep_scrub_stamp":"2024-09-13T16:48:51.551365+0000","last_clean_scrub_stamp":"2024-09-13T16:48:51.551365+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:08:42.165482+0000","scrub_duration":0.002414116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":278528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"22'4","reported_seq":31,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:36.093320+0000","last_change":"2024-09-13T16:48:36.093416+0000","last_active":"2024-09-13T16:48:36.093320+0000","last_peered":"2024-09-13T16:48:36.093320+0000","last_clean":"2024-09-13T16:48:36.093320+0000","last_became_active":"2024-09-13T16:43:54.559353+0000","last_became_peered":"2024-09-13T16:43:54.559353+0000","last_unstale":"2024-09-13T16:48:36.093320+0000","last_undegraded":"2024-09-13T16:48:36.093320+0000","last_fullsized":"2024-09-13T16:48:36.093320+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'4","last_scrub_stamp":"2024-09-13T16:48:36.093289+0000","last_deep_scrub":"22'4","last_deep_scrub_stamp":"2024-09-13T16:48:36.093289+0000","last_clean_scrub_stamp":"2024-09-13T16:48:36.093289+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:27.023954+0000","scrub_duration":0.0036442430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"22'209","reported_seq":235,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:53.629438+0000","last_change":"2024-09-13T16:48:53.629532+0000","last_active":"2024-09-13T16:48:53.629438+0000","last_peered":"2024-09-13T16:48:53.629438+0000","last_clean":"2024-09-13T16:48:53.629438+0000","last_became_active":"2024-09-13T16:43:54.992249+0000","last_became_peered":"2024-09-13T16:43:54.992249+0000","last_unstale":"2024-09-13T16:48:53.629438+0000","last_undegraded":"2024-09-13T16:48:53.629438+0000","last_fullsized":"2024-09-13T16:48:53.629438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'209","last_scrub_stamp":"2024-09-13T16:48:53.629413+0000","last_deep_scrub":"22'209","last_deep_scrub_stamp":"2024-09-13T16:48:53.629413+0000","last_clean_scrub_stamp":"2024-09-13T16:48:53.629413+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:06.071346+0000","scrub_duration":0.0022279719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'8","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:30.390613+0000","last_change":"2024-09-13T16:48:30.390713+0000","last_active":"2024-09-13T16:48:30.390613+0000","last_peered":"2024-09-13T16:48:30.390613+0000","last_clean":"2024-09-13T16:48:30.390613+0000","last_became_active":"2024-09-13T16:43:54.547074+0000","last_became_peered":"2024-09-13T16:43:54.547074+0000","last_unstale":"2024-09-13T16:48:30.390613+0000","last_undegraded":"2024-09-13T16:48:30.390613+0000","last_fullsized":"2024-09-13T16:48:30.390613+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'8","last_scrub_stamp":"2024-09-13T16:48:30.390596+0000","last_deep_scrub":"22'8","last_deep_scrub_stamp":"2024-09-13T16:48:30.390596+0000","last_clean_scrub_stamp":"2024-09-13T16:48:30.390596+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:11.620523+0000","scrub_duration":0.0040624839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"22'172","reported_seq":198,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:54.931872+0000","last_change":"2024-09-13T16:48:54.931963+0000","last_active":"2024-09-13T16:48:54.931872+0000","last_peered":"2024-09-13T16:48:54.931872+0000","last_clean":"2024-09-13T16:48:54.931872+0000","last_became_active":"2024-09-13T16:43:54.990196+0000","last_became_peered":"2024-09-13T16:43:54.990196+0000","last_unstale":"2024-09-13T16:48:54.931872+0000","last_undegraded":"2024-09-13T16:48:54.931872+0000","last_fullsized":"2024-09-13T16:48:54.931872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'172","last_scrub_stamp":"2024-09-13T16:48:54.931850+0000","last_deep_scrub":"22'172","last_deep_scrub_stamp":"2024-09-13T16:48:54.931850+0000","last_clean_scrub_stamp":"2024-09-13T16:48:54.931850+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:07.373783+0000","scrub_duration":0.001519357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'119","reported_seq":182,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:35.961536+0000","last_change":"2024-09-13T16:48:35.961673+0000","last_active":"2024-09-13T16:48:35.961536+0000","last_peered":"2024-09-13T16:48:35.961536+0000","last_clean":"2024-09-13T16:48:35.961536+0000","last_became_active":"2024-09-13T16:43:51.913797+0000","last_became_peered":"2024-09-13T16:43:51.913797+0000","last_unstale":"2024-09-13T16:48:35.961536+0000","last_undegraded":"2024-09-13T16:48:35.961536+0000","last_fullsized":"2024-09-13T16:48:35.961536+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'119","last_scrub_stamp":"2024-09-13T16:48:35.961513+0000","last_deep_scrub":"22'119","last_deep_scrub_stamp":"2024-09-13T16:48:35.961513+0000","last_clean_scrub_stamp":"2024-09-13T16:48:35.961513+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:59:32.930667+0000","scrub_duration":0.0053261610000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"22'11","reported_seq":38,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:40.552690+0000","last_change":"2024-09-13T16:48:40.552797+0000","last_active":"2024-09-13T16:48:40.552690+0000","last_peered":"2024-09-13T16:48:40.552690+0000","last_clean":"2024-09-13T16:48:40.552690+0000","last_became_active":"2024-09-13T16:43:54.563949+0000","last_became_peered":"2024-09-13T16:43:54.563949+0000","last_unstale":"2024-09-13T16:48:40.552690+0000","last_undegraded":"2024-09-13T16:48:40.552690+0000","last_fullsized":"2024-09-13T16:48:40.552690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'11","last_scrub_stamp":"2024-09-13T16:48:40.552667+0000","last_deep_scrub":"22'11","last_deep_scrub_stamp":"2024-09-13T16:48:40.552667+0000","last_clean_scrub_stamp":"2024-09-13T16:48:40.552667+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:19.643388+0000","scrub_duration":0.004950004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"22'215","reported_seq":239,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:48.528169+0000","last_change":"2024-09-13T16:48:48.528169+0000","last_active":"2024-09-13T16:48:48.528169+0000","last_peered":"2024-09-13T16:48:48.528169+0000","last_clean":"2024-09-13T16:48:48.528169+0000","last_became_active":"2024-09-13T16:44:59.386756+0000","last_became_peered":"2024-09-13T16:44:59.386756+0000","last_unstale":"2024-09-13T16:48:48.528169+0000","last_undegraded":"2024-09-13T16:48:48.528169+0000","last_fullsized":"2024-09-13T16:48:48.528169+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'215","last_scrub_stamp":"2024-09-13T16:48:48.528033+0000","last_deep_scrub":"22'215","last_deep_scrub_stamp":"2024-09-13T16:48:48.528033+0000","last_clean_scrub_stamp":"2024-09-13T16:48:48.528033+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:27.618755+0000","scrub_duration":0.0021049049999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":186,"num_write_kb":507904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"22'5","reported_seq":32,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:37.992778+0000","last_change":"2024-09-13T16:48:37.992883+0000","last_active":"2024-09-13T16:48:37.992778+0000","last_peered":"2024-09-13T16:48:37.992778+0000","last_clean":"2024-09-13T16:48:37.992778+0000","last_became_active":"2024-09-13T16:43:54.564605+0000","last_became_peered":"2024-09-13T16:43:54.564605+0000","last_unstale":"2024-09-13T16:48:37.992778+0000","last_undegraded":"2024-09-13T16:48:37.992778+0000","last_fullsized":"2024-09-13T16:48:37.992778+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'5","last_scrub_stamp":"2024-09-13T16:48:37.992756+0000","last_deep_scrub":"22'5","last_deep_scrub_stamp":"2024-09-13T16:48:37.992756+0000","last_clean_scrub_stamp":"2024-09-13T16:48:37.992756+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:28.923420+0000","scrub_duration":0.0036587429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"22'192","reported_seq":218,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:33.451090+0000","last_change":"2024-09-13T16:48:33.451217+0000","last_active":"2024-09-13T16:48:33.451090+0000","last_peered":"2024-09-13T16:48:33.451090+0000","last_clean":"2024-09-13T16:48:33.451090+0000","last_became_active":"2024-09-13T16:43:54.997510+0000","last_became_peered":"2024-09-13T16:43:54.997510+0000","last_unstale":"2024-09-13T16:48:33.451090+0000","last_undegraded":"2024-09-13T16:48:33.451090+0000","last_fullsized":"2024-09-13T16:48:33.451090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'192","last_scrub_stamp":"2024-09-13T16:48:33.451068+0000","last_deep_scrub":"22'192","last_deep_scrub_stamp":"2024-09-13T16:48:33.451068+0000","last_clean_scrub_stamp":"2024-09-13T16:48:33.451068+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:09:56.818020+0000","scrub_duration":0.0021645589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'15","reported_seq":47,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:36.404304+0000","last_change":"2024-09-13T16:48:36.404425+0000","last_active":"2024-09-13T16:48:36.404304+0000","last_peered":"2024-09-13T16:48:36.404304+0000","last_clean":"2024-09-13T16:48:36.404304+0000","last_became_active":"2024-09-13T16:43:54.566023+0000","last_became_peered":"2024-09-13T16:43:54.566023+0000","last_unstale":"2024-09-13T16:48:36.404304+0000","last_undegraded":"2024-09-13T16:48:36.404304+0000","last_fullsized":"2024-09-13T16:48:36.404304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-09-13T16:48:36.404278+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-09-13T16:48:36.404278+0000","last_clean_scrub_stamp":"2024-09-13T16:48:36.404278+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:09:59.771230+0000","scrub_duration":0.0040041060000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"22'167","reported_seq":193,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:45.035817+0000","last_change":"2024-09-13T16:48:45.035919+0000","last_active":"2024-09-13T16:48:45.035817+0000","last_peered":"2024-09-13T16:48:45.035817+0000","last_clean":"2024-09-13T16:48:45.035817+0000","last_became_active":"2024-09-13T16:43:54.980766+0000","last_became_peered":"2024-09-13T16:43:54.980766+0000","last_unstale":"2024-09-13T16:48:45.035817+0000","last_undegraded":"2024-09-13T16:48:45.035817+0000","last_fullsized":"2024-09-13T16:48:45.035817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'167","last_scrub_stamp":"2024-09-13T16:48:45.035795+0000","last_deep_scrub":"22'167","last_deep_scrub_stamp":"2024-09-13T16:48:45.035795+0000","last_clean_scrub_stamp":"2024-09-13T16:48:45.035795+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:09.803222+0000","scrub_duration":0.001815189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"22'7","reported_seq":34,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:32.563054+0000","last_change":"2024-09-13T16:48:32.563164+0000","last_active":"2024-09-13T16:48:32.563054+0000","last_peered":"2024-09-13T16:48:32.563054+0000","last_clean":"2024-09-13T16:48:32.563054+0000","last_became_active":"2024-09-13T16:43:54.563522+0000","last_became_peered":"2024-09-13T16:43:54.563522+0000","last_unstale":"2024-09-13T16:48:32.563054+0000","last_undegraded":"2024-09-13T16:48:32.563054+0000","last_fullsized":"2024-09-13T16:48:32.563054+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'7","last_scrub_stamp":"2024-09-13T16:48:32.563029+0000","last_deep_scrub":"22'7","last_deep_scrub_stamp":"2024-09-13T16:48:32.563029+0000","last_clean_scrub_stamp":"2024-09-13T16:48:32.563029+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:21.462253+0000","scrub_duration":0.0038373040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"22'163","reported_seq":189,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:30.616538+0000","last_change":"2024-09-13T16:48:30.616641+0000","last_active":"2024-09-13T16:48:30.616538+0000","last_peered":"2024-09-13T16:48:30.616538+0000","last_clean":"2024-09-13T16:48:30.616538+0000","last_became_active":"2024-09-13T16:43:54.998803+0000","last_became_peered":"2024-09-13T16:43:54.998803+0000","last_unstale":"2024-09-13T16:48:30.616538+0000","last_undegraded":"2024-09-13T16:48:30.616538+0000","last_fullsized":"2024-09-13T16:48:30.616538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'163","last_scrub_stamp":"2024-09-13T16:48:30.616512+0000","last_deep_scrub":"22'163","last_deep_scrub_stamp":"2024-09-13T16:48:30.616512+0000","last_clean_scrub_stamp":"2024-09-13T16:48:30.616512+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:33:30.274099+0000","scrub_duration":0.0019863279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":28,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:40.074852+0000","last_change":"2024-09-13T16:48:40.075007+0000","last_active":"2024-09-13T16:48:40.074852+0000","last_peered":"2024-09-13T16:48:40.074852+0000","last_clean":"2024-09-13T16:48:40.074852+0000","last_became_active":"2024-09-13T16:43:54.568846+0000","last_became_peered":"2024-09-13T16:43:54.568846+0000","last_unstale":"2024-09-13T16:48:40.074852+0000","last_undegraded":"2024-09-13T16:48:40.074852+0000","last_fullsized":"2024-09-13T16:48:40.074852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'1","last_scrub_stamp":"2024-09-13T16:48:40.074829+0000","last_deep_scrub":"20'1","last_deep_scrub_stamp":"2024-09-13T16:48:40.074829+0000","last_clean_scrub_stamp":"2024-09-13T16:48:40.074829+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:36:47.384962+0000","scrub_duration":0.001876604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"22'179","reported_seq":205,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:46.643014+0000","last_change":"2024-09-13T16:48:46.643116+0000","last_active":"2024-09-13T16:48:46.643014+0000","last_peered":"2024-09-13T16:48:46.643014+0000","last_clean":"2024-09-13T16:48:46.643014+0000","last_became_active":"2024-09-13T16:43:54.991659+0000","last_became_peered":"2024-09-13T16:43:54.991659+0000","last_unstale":"2024-09-13T16:48:46.643014+0000","last_undegraded":"2024-09-13T16:48:46.643014+0000","last_fullsized":"2024-09-13T16:48:46.643014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'179","last_scrub_stamp":"2024-09-13T16:48:46.642991+0000","last_deep_scrub":"22'179","last_deep_scrub_stamp":"2024-09-13T16:48:46.642991+0000","last_clean_scrub_stamp":"2024-09-13T16:48:46.642991+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:48:01.008557+0000","scrub_duration":0.0022816239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"22'7","reported_seq":34,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:44.922669+0000","last_change":"2024-09-13T16:48:44.922788+0000","last_active":"2024-09-13T16:48:44.922669+0000","last_peered":"2024-09-13T16:48:44.922669+0000","last_clean":"2024-09-13T16:48:44.922669+0000","last_became_active":"2024-09-13T16:43:54.568903+0000","last_became_peered":"2024-09-13T16:43:54.568903+0000","last_unstale":"2024-09-13T16:48:44.922669+0000","last_undegraded":"2024-09-13T16:48:44.922669+0000","last_fullsized":"2024-09-13T16:48:44.922669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'7","last_scrub_stamp":"2024-09-13T16:48:44.922644+0000","last_deep_scrub":"22'7","last_deep_scrub_stamp":"2024-09-13T16:48:44.922644+0000","last_clean_scrub_stamp":"2024-09-13T16:48:44.922644+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:24.013365+0000","scrub_duration":0.0046489829999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"22'206","reported_seq":232,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:47.016450+0000","last_change":"2024-09-13T16:48:47.016564+0000","last_active":"2024-09-13T16:48:47.016450+0000","last_peered":"2024-09-13T16:48:47.016450+0000","last_clean":"2024-09-13T16:48:47.016450+0000","last_became_active":"2024-09-13T16:43:54.979584+0000","last_became_peered":"2024-09-13T16:43:54.979584+0000","last_unstale":"2024-09-13T16:48:47.016450+0000","last_undegraded":"2024-09-13T16:48:47.016450+0000","last_fullsized":"2024-09-13T16:48:47.016450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'206","last_scrub_stamp":"2024-09-13T16:48:47.016426+0000","last_deep_scrub":"22'206","last_deep_scrub_stamp":"2024-09-13T16:48:47.016426+0000","last_clean_scrub_stamp":"2024-09-13T16:48:47.016426+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:14.765912+0000","scrub_duration":0.001764435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'24","reported_seq":51,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:34.077498+0000","last_change":"2024-09-13T16:48:34.077608+0000","last_active":"2024-09-13T16:48:34.077498+0000","last_peered":"2024-09-13T16:48:34.077498+0000","last_clean":"2024-09-13T16:48:34.077498+0000","last_became_active":"2024-09-13T16:43:54.560030+0000","last_became_peered":"2024-09-13T16:43:54.560030+0000","last_unstale":"2024-09-13T16:48:34.077498+0000","last_undegraded":"2024-09-13T16:48:34.077498+0000","last_fullsized":"2024-09-13T16:48:34.077498+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-09-13T16:48:34.077471+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-09-13T16:48:34.077471+0000","last_clean_scrub_stamp":"2024-09-13T16:48:34.077471+0000","objects_scrubbed":4,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:59:31.046625+0000","scrub_duration":0.003834367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'196","reported_seq":222,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:43.939466+0000","last_change":"2024-09-13T16:48:43.939576+0000","last_active":"2024-09-13T16:48:43.939466+0000","last_peered":"2024-09-13T16:48:43.939466+0000","last_clean":"2024-09-13T16:48:43.939466+0000","last_became_active":"2024-09-13T16:43:54.990440+0000","last_became_peered":"2024-09-13T16:43:54.990440+0000","last_unstale":"2024-09-13T16:48:43.939466+0000","last_undegraded":"2024-09-13T16:48:43.939466+0000","last_fullsized":"2024-09-13T16:48:43.939466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'196","last_scrub_stamp":"2024-09-13T16:48:43.939446+0000","last_deep_scrub":"22'196","last_deep_scrub_stamp":"2024-09-13T16:48:43.939446+0000","last_clean_scrub_stamp":"2024-09-13T16:48:43.939446+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:23.030168+0000","scrub_duration":0.002075665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'12","reported_seq":39,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:29.809769+0000","last_change":"2024-09-13T16:48:29.809876+0000","last_active":"2024-09-13T16:48:29.809769+0000","last_peered":"2024-09-13T16:48:29.809769+0000","last_clean":"2024-09-13T16:48:29.809769+0000","last_became_active":"2024-09-13T16:43:54.558194+0000","last_became_peered":"2024-09-13T16:43:54.558194+0000","last_unstale":"2024-09-13T16:48:29.809769+0000","last_undegraded":"2024-09-13T16:48:29.809769+0000","last_fullsized":"2024-09-13T16:48:29.809769+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-09-13T16:48:29.809742+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-09-13T16:48:29.809742+0000","last_clean_scrub_stamp":"2024-09-13T16:48:29.809742+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:33:29.467329+0000","scrub_duration":0.0040895360000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"22'161","reported_seq":187,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:37.486307+0000","last_change":"2024-09-13T16:48:37.486418+0000","last_active":"2024-09-13T16:48:37.486307+0000","last_peered":"2024-09-13T16:48:37.486307+0000","last_clean":"2024-09-13T16:48:37.486307+0000","last_became_active":"2024-09-13T16:43:54.995397+0000","last_became_peered":"2024-09-13T16:43:54.995397+0000","last_unstale":"2024-09-13T16:48:37.486307+0000","last_undegraded":"2024-09-13T16:48:37.486307+0000","last_fullsized":"2024-09-13T16:48:37.486307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'161","last_scrub_stamp":"2024-09-13T16:48:37.486283+0000","last_deep_scrub":"22'161","last_deep_scrub_stamp":"2024-09-13T16:48:37.486283+0000","last_clean_scrub_stamp":"2024-09-13T16:48:37.486283+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:26.385507+0000","scrub_duration":0.0018563010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"22'3","reported_seq":30,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:47.903413+0000","last_change":"2024-09-13T16:48:47.903511+0000","last_active":"2024-09-13T16:48:47.903413+0000","last_peered":"2024-09-13T16:48:47.903413+0000","last_clean":"2024-09-13T16:48:47.903413+0000","last_became_active":"2024-09-13T16:43:54.568892+0000","last_became_peered":"2024-09-13T16:43:54.568892+0000","last_unstale":"2024-09-13T16:48:47.903413+0000","last_undegraded":"2024-09-13T16:48:47.903413+0000","last_fullsized":"2024-09-13T16:48:47.903413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-09-13T16:48:47.903389+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-09-13T16:48:47.903389+0000","last_clean_scrub_stamp":"2024-09-13T16:48:47.903389+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:02:49.388871+0000","scrub_duration":0.0037439740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"22'198","reported_seq":224,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:31.599065+0000","last_change":"2024-09-13T16:48:31.599179+0000","last_active":"2024-09-13T16:48:31.599065+0000","last_peered":"2024-09-13T16:48:31.599065+0000","last_clean":"2024-09-13T16:48:31.599065+0000","last_became_active":"2024-09-13T16:43:54.983886+0000","last_became_peered":"2024-09-13T16:43:54.983886+0000","last_unstale":"2024-09-13T16:48:31.599065+0000","last_undegraded":"2024-09-13T16:48:31.599065+0000","last_fullsized":"2024-09-13T16:48:31.599065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'198","last_scrub_stamp":"2024-09-13T16:48:31.599041+0000","last_deep_scrub":"22'198","last_deep_scrub_stamp":"2024-09-13T16:48:31.599041+0000","last_clean_scrub_stamp":"2024-09-13T16:48:31.599041+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:55:15.644305+0000","scrub_duration":0.0022504479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"22'11","reported_seq":38,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:32.092037+0000","last_change":"2024-09-13T16:48:32.092150+0000","last_active":"2024-09-13T16:48:32.092037+0000","last_peered":"2024-09-13T16:48:32.092037+0000","last_clean":"2024-09-13T16:48:32.092037+0000","last_became_active":"2024-09-13T16:43:54.554230+0000","last_became_peered":"2024-09-13T16:43:54.554230+0000","last_unstale":"2024-09-13T16:48:32.092037+0000","last_undegraded":"2024-09-13T16:48:32.092037+0000","last_fullsized":"2024-09-13T16:48:32.092037+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'11","last_scrub_stamp":"2024-09-13T16:48:32.092012+0000","last_deep_scrub":"22'11","last_deep_scrub_stamp":"2024-09-13T16:48:32.092012+0000","last_clean_scrub_stamp":"2024-09-13T16:48:32.092012+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:08.810087+0000","scrub_duration":0.0049653240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"22'178","reported_seq":204,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:48.902978+0000","last_change":"2024-09-13T16:48:48.903084+0000","last_active":"2024-09-13T16:48:48.902978+0000","last_peered":"2024-09-13T16:48:48.902978+0000","last_clean":"2024-09-13T16:48:48.902978+0000","last_became_active":"2024-09-13T16:43:54.984221+0000","last_became_peered":"2024-09-13T16:43:54.984221+0000","last_unstale":"2024-09-13T16:48:48.902978+0000","last_undegraded":"2024-09-13T16:48:48.902978+0000","last_fullsized":"2024-09-13T16:48:48.902978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'178","last_scrub_stamp":"2024-09-13T16:48:48.902943+0000","last_deep_scrub":"22'178","last_deep_scrub_stamp":"2024-09-13T16:48:48.902943+0000","last_clean_scrub_stamp":"2024-09-13T16:48:48.902943+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:06.299355+0000","scrub_duration":0.001721457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"22'5","reported_seq":32,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:35.116763+0000","last_change":"2024-09-13T16:48:35.116864+0000","last_active":"2024-09-13T16:48:35.116763+0000","last_peered":"2024-09-13T16:48:35.116763+0000","last_clean":"2024-09-13T16:48:35.116763+0000","last_became_active":"2024-09-13T16:43:54.556888+0000","last_became_peered":"2024-09-13T16:43:54.556888+0000","last_unstale":"2024-09-13T16:48:35.116763+0000","last_undegraded":"2024-09-13T16:48:35.116763+0000","last_fullsized":"2024-09-13T16:48:35.116763+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'5","last_scrub_stamp":"2024-09-13T16:48:35.116742+0000","last_deep_scrub":"22'5","last_deep_scrub_stamp":"2024-09-13T16:48:35.116742+0000","last_clean_scrub_stamp":"2024-09-13T16:48:35.116742+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:05.443913+0000","scrub_duration":0.0036882849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"22'217","reported_seq":243,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:35.884068+0000","last_change":"2024-09-13T16:48:35.884174+0000","last_active":"2024-09-13T16:48:35.884068+0000","last_peered":"2024-09-13T16:48:35.884068+0000","last_clean":"2024-09-13T16:48:35.884068+0000","last_became_active":"2024-09-13T16:43:54.999101+0000","last_became_peered":"2024-09-13T16:43:54.999101+0000","last_unstale":"2024-09-13T16:48:35.884068+0000","last_undegraded":"2024-09-13T16:48:35.884068+0000","last_fullsized":"2024-09-13T16:48:35.884068+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'217","last_scrub_stamp":"2024-09-13T16:48:35.884045+0000","last_deep_scrub":"22'217","last_deep_scrub_stamp":"2024-09-13T16:48:35.884045+0000","last_clean_scrub_stamp":"2024-09-13T16:48:35.884045+0000","objects_scrubbed":0,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:24.783269+0000","scrub_duration":0.0018987959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'7","reported_seq":34,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:41.928025+0000","last_change":"2024-09-13T16:48:41.928129+0000","last_active":"2024-09-13T16:48:41.928025+0000","last_peered":"2024-09-13T16:48:41.928025+0000","last_clean":"2024-09-13T16:48:41.928025+0000","last_became_active":"2024-09-13T16:43:54.558233+0000","last_became_peered":"2024-09-13T16:43:54.558233+0000","last_unstale":"2024-09-13T16:48:41.928025+0000","last_undegraded":"2024-09-13T16:48:41.928025+0000","last_fullsized":"2024-09-13T16:48:41.928025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'7","last_scrub_stamp":"2024-09-13T16:48:41.927998+0000","last_deep_scrub":"22'7","last_deep_scrub_stamp":"2024-09-13T16:48:41.927998+0000","last_clean_scrub_stamp":"2024-09-13T16:48:41.927998+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:36:49.238131+0000","scrub_duration":0.0029359780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"22'190","reported_seq":216,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:53.599896+0000","last_change":"2024-09-13T16:48:53.600016+0000","last_active":"2024-09-13T16:48:53.599896+0000","last_peered":"2024-09-13T16:48:53.599896+0000","last_clean":"2024-09-13T16:48:53.599896+0000","last_became_active":"2024-09-13T16:43:54.986084+0000","last_became_peered":"2024-09-13T16:43:54.986084+0000","last_unstale":"2024-09-13T16:48:53.599896+0000","last_undegraded":"2024-09-13T16:48:53.599896+0000","last_fullsized":"2024-09-13T16:48:53.599896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'190","last_scrub_stamp":"2024-09-13T16:48:53.599872+0000","last_deep_scrub":"22'190","last_deep_scrub_stamp":"2024-09-13T16:48:53.599872+0000","last_clean_scrub_stamp":"2024-09-13T16:48:53.599872+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:18.367299+0000","scrub_duration":0.0016158990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"22'165","reported_seq":191,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:51.660922+0000","last_change":"2024-09-13T16:48:51.660922+0000","last_active":"2024-09-13T16:48:51.660922+0000","last_peered":"2024-09-13T16:48:51.660922+0000","last_clean":"2024-09-13T16:48:51.660922+0000","last_became_active":"2024-09-13T16:43:54.982338+0000","last_became_peered":"2024-09-13T16:43:54.982338+0000","last_unstale":"2024-09-13T16:48:51.660922+0000","last_undegraded":"2024-09-13T16:48:51.660922+0000","last_fullsized":"2024-09-13T16:48:51.660922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'165","last_scrub_stamp":"2024-09-13T16:48:51.660800+0000","last_deep_scrub":"22'165","last_deep_scrub_stamp":"2024-09-13T16:48:51.660800+0000","last_clean_scrub_stamp":"2024-09-13T16:48:51.660800+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:51.051224+0000","scrub_duration":0.00170703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'8","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:36.989486+0000","last_change":"2024-09-13T16:48:36.989486+0000","last_active":"2024-09-13T16:48:36.989486+0000","last_peered":"2024-09-13T16:48:36.989486+0000","last_clean":"2024-09-13T16:48:36.989486+0000","last_became_active":"2024-09-13T16:43:54.558332+0000","last_became_peered":"2024-09-13T16:43:54.558332+0000","last_unstale":"2024-09-13T16:48:36.989486+0000","last_undegraded":"2024-09-13T16:48:36.989486+0000","last_fullsized":"2024-09-13T16:48:36.989486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'8","last_scrub_stamp":"2024-09-13T16:48:36.989352+0000","last_deep_scrub":"22'8","last_deep_scrub_stamp":"2024-09-13T16:48:36.989352+0000","last_clean_scrub_stamp":"2024-09-13T16:48:36.989352+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:07.316522+0000","scrub_duration":0.0034627450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'152","reported_seq":175,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:54.088485+0000","last_change":"2024-09-13T16:48:54.088588+0000","last_active":"2024-09-13T16:48:54.088485+0000","last_peered":"2024-09-13T16:48:54.088485+0000","last_clean":"2024-09-13T16:48:54.088485+0000","last_became_active":"2024-09-13T16:44:58.957568+0000","last_became_peered":"2024-09-13T16:44:58.957568+0000","last_unstale":"2024-09-13T16:48:54.088485+0000","last_undegraded":"2024-09-13T16:48:54.088485+0000","last_fullsized":"2024-09-13T16:48:54.088485+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'152","last_scrub_stamp":"2024-09-13T16:48:54.088462+0000","last_deep_scrub":"22'152","last_deep_scrub_stamp":"2024-09-13T16:48:54.088462+0000","last_clean_scrub_stamp":"2024-09-13T16:48:54.088462+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:31.426849+0000","scrub_duration":0.0020483239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'20","reported_seq":47,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:46.886655+0000","last_change":"2024-09-13T16:48:46.886655+0000","last_active":"2024-09-13T16:48:46.886655+0000","last_peered":"2024-09-13T16:48:46.886655+0000","last_clean":"2024-09-13T16:48:46.886655+0000","last_became_active":"2024-09-13T16:43:54.559290+0000","last_became_peered":"2024-09-13T16:43:54.559290+0000","last_unstale":"2024-09-13T16:48:46.886655+0000","last_undegraded":"2024-09-13T16:48:46.886655+0000","last_fullsized":"2024-09-13T16:48:46.886655+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'20","last_scrub_stamp":"2024-09-13T16:48:46.886511+0000","last_deep_scrub":"22'20","last_deep_scrub_stamp":"2024-09-13T16:48:46.886511+0000","last_clean_scrub_stamp":"2024-09-13T16:48:46.886511+0000","objects_scrubbed":7,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:11.653937+0000","scrub_duration":0.0070396130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"22'180","reported_seq":206,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:51.535400+0000","last_change":"2024-09-13T16:48:51.535496+0000","last_active":"2024-09-13T16:48:51.535400+0000","last_peered":"2024-09-13T16:48:51.535400+0000","last_clean":"2024-09-13T16:48:51.535400+0000","last_became_active":"2024-09-13T16:43:55.000196+0000","last_became_peered":"2024-09-13T16:43:55.000196+0000","last_unstale":"2024-09-13T16:48:51.535400+0000","last_undegraded":"2024-09-13T16:48:51.535400+0000","last_fullsized":"2024-09-13T16:48:51.535400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'180","last_scrub_stamp":"2024-09-13T16:48:51.535377+0000","last_deep_scrub":"22'180","last_deep_scrub_stamp":"2024-09-13T16:48:51.535377+0000","last_clean_scrub_stamp":"2024-09-13T16:48:51.535377+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:48:05.900943+0000","scrub_duration":0.001971244,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":29,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:31.845619+0000","last_change":"2024-09-13T16:48:31.845740+0000","last_active":"2024-09-13T16:48:31.845619+0000","last_peered":"2024-09-13T16:48:31.845619+0000","last_clean":"2024-09-13T16:48:31.845619+0000","last_became_active":"2024-09-13T16:43:54.559586+0000","last_became_peered":"2024-09-13T16:43:54.559586+0000","last_unstale":"2024-09-13T16:48:31.845619+0000","last_undegraded":"2024-09-13T16:48:31.845619+0000","last_fullsized":"2024-09-13T16:48:31.845619+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'2","last_scrub_stamp":"2024-09-13T16:48:31.845590+0000","last_deep_scrub":"20'2","last_deep_scrub_stamp":"2024-09-13T16:48:31.845590+0000","last_clean_scrub_stamp":"2024-09-13T16:48:31.845590+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:09:55.212541+0000","scrub_duration":0.0019454159999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"22'152","reported_seq":178,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:41.886790+0000","last_change":"2024-09-13T16:48:41.886897+0000","last_active":"2024-09-13T16:48:41.886790+0000","last_peered":"2024-09-13T16:48:41.886790+0000","last_clean":"2024-09-13T16:48:41.886790+0000","last_became_active":"2024-09-13T16:43:54.995802+0000","last_became_peered":"2024-09-13T16:43:54.995802+0000","last_unstale":"2024-09-13T16:48:41.886790+0000","last_undegraded":"2024-09-13T16:48:41.886790+0000","last_fullsized":"2024-09-13T16:48:41.886790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'152","last_scrub_stamp":"2024-09-13T16:48:41.886768+0000","last_deep_scrub":"22'152","last_deep_scrub_stamp":"2024-09-13T16:48:41.886768+0000","last_clean_scrub_stamp":"2024-09-13T16:48:41.886768+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:39.898801+0000","scrub_duration":0.00231227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'19","reported_seq":46,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-13T16:48:27.429267+0000","last_change":"2024-09-13T16:48:27.429354+0000","last_active":"2024-09-13T16:48:27.429267+0000","last_peered":"2024-09-13T16:48:27.429267+0000","last_clean":"2024-09-13T16:48:27.429267+0000","last_became_active":"2024-09-13T16:43:54.560552+0000","last_became_peered":"2024-09-13T16:43:54.560552+0000","last_unstale":"2024-09-13T16:48:27.429267+0000","last_undegraded":"2024-09-13T16:48:27.429267+0000","last_fullsized":"2024-09-13T16:48:27.429267+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'19","last_scrub_stamp":"2024-09-13T16:48:27.429247+0000","last_deep_scrub":"22'19","last_deep_scrub_stamp":"2024-09-13T16:48:27.429247+0000","last_clean_scrub_stamp":"2024-09-13T16:48:27.429247+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:55:11.474511+0000","scrub_duration":0.0049610629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11253,"ondisk_log_size":11253,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":371833,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":410,"num_write_kb":567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":743666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":465,"ondisk_log_size":465,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65086,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39628,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332212,"statfs":{"total":96636764160,"available":96596185088,"internally_reserved":0,"allocated":331776,"data_stored":675500,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":40237514},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59699999999999998,"15min":0.57499999999999996},"min":{"1min":0.378,"5min":0.245,"15min":0.245},"max":{"1min":0.89900000000000002,"5min":5.0620000000000003,"15min":5.0620000000000003},"last":0.374},{"interface":"front","average":{"1min":0.621,"5min":0.69799999999999995,"15min":0.60499999999999998},"min":{"1min":0.46800000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.79900000000000004,"5min":5.1859999999999999,"15min":5.1859999999999999},"last":0.42299999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.56299999999999994,"15min":0.56200000000000006},"min":{"1min":0.46300000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.76900000000000002,"5min":1.716,"15min":1.716},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.629,"15min":0.59399999999999997},"min":{"1min":0.40500000000000003,"5min":0.314,"15min":0.314},"max":{"1min":0.84299999999999997,"5min":3.6070000000000002,"15min":3.6070000000000002},"last":0.50600000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.58699999999999997,"15min":0.58799999999999997},"min":{"1min":0.437,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77000000000000002,"5min":2.4929999999999999,"15min":2.4929999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64200000000000002,"15min":0.61899999999999999},"min":{"1min":0.51200000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.91400000000000003,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":0.57199999999999995}]},{"osd":3,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.57099999999999995,"15min":0.58299999999999996},"min":{"1min":0.44900000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.78200000000000003,"5min":3.8490000000000002,"15min":3.8490000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66400000000000003,"15min":0.60999999999999999},"min":{"1min":0.53200000000000003,"5min":0.372,"15min":0.372},"max":{"1min":0.81200000000000006,"5min":4.0129999999999999,"15min":4.0129999999999999},"last":0.58799999999999997}]},{"osd":4,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.435,"15min":0.443},"min":{"1min":0.35899999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.67600000000000005,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.437,"15min":0.48699999999999999},"min":{"1min":0.33100000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61699999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.39300000000000002}]},{"osd":5,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.45800000000000002,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.68999999999999995,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.497,"15min":0.52100000000000002},"min":{"1min":0.372,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76700000000000002,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.53400000000000003}]},{"osd":6,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52500000000000002,"15min":0.56000000000000005},"min":{"1min":0.42499999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.72999999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.498,"15min":0.53800000000000003},"min":{"1min":0.40799999999999997,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70199999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.52000000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35812,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336028,"statfs":{"total":96636764160,"available":96600092672,"internally_reserved":0,"allocated":618496,"data_stored":448920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":36043209},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.69899999999999995,"15min":0.627},"min":{"1min":0.28899999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.82399999999999995,"5min":8.468,"15min":8.468},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.67400000000000004,"15min":0.625},"min":{"1min":0.40000000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82099999999999995,"5min":7.5739999999999998,"15min":7.5739999999999998},"last":0.60199999999999998}]},{"osd":1,"last update":"Fri Sep 13 16:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.46000000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.86299999999999999,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.629},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.60399999999999998,"15min":0.63800000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83999999999999997,"5min":1.143,"15min":1.143},"last":0.65100000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60399999999999998,"15min":0.622},"min":{"1min":0.433,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.85999999999999999,"5min":2.431,"15min":2.431},"last":0.503},{"interface":"front","average":{"1min":0.626,"5min":0.79600000000000004,"15min":0.64500000000000002},"min":{"1min":0.42999999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.84999999999999998,"5min":6.9139999999999997,"15min":6.9139999999999997},"last":0.63900000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.67800000000000005,"15min":0.61199999999999999},"min":{"1min":0.375,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.84699999999999998,"5min":6.2400000000000002,"15min":6.2400000000000002},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.70799999999999996,"15min":0.66200000000000003},"min":{"1min":0.32100000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.86699999999999999,"5min":3.4500000000000002,"15min":3.4500000000000002},"last":0.58499999999999996}]},{"osd":4,"last update":"Fri Sep 13 16:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.40000000000000002,"15min":0.44700000000000001},"min":{"1min":0.29799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.65100000000000002,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.42599999999999999,"15min":0.47899999999999998},"min":{"1min":0.32800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.66300000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.44500000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.47699999999999998,"15min":0.52500000000000002},"min":{"1min":0.35799999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.72399999999999998,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.501,"5min":0.49199999999999999,"15min":0.55800000000000005},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68999999999999995,"5min":0.752,"15min":0.752},"last":0.52900000000000003}]},{"osd":7,"last update":"Fri Sep 13 16:48:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.47099999999999997,"15min":0.51600000000000001},"min":{"1min":0.372,"5min":0.253,"15min":0.253},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52600000000000002,"15min":0.57599999999999996},"min":{"1min":0.39700000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.79300000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.61299999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574913,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39572,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332268,"statfs":{"total":96636764160,"available":96596242432,"internally_reserved":0,"allocated":274432,"data_stored":86302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.58199999999999996,"15min":0.56299999999999994},"min":{"1min":0.45400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68000000000000005,"5min":3.411,"15min":3.411},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59599999999999997,"15min":0.60099999999999998},"min":{"1min":0.35699999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.80500000000000005,"5min":3.117,"15min":3.117},"last":0.70899999999999996}]},{"osd":1,"last update":"Fri Sep 13 16:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.42299999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.86599999999999999,"5min":1.502,"15min":1.502},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.64800000000000002,"15min":0.59099999999999997},"min":{"1min":0.502,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72099999999999997,"5min":4.3860000000000001,"15min":4.3860000000000001},"last":0.69799999999999995}]},{"osd":2,"last update":"Fri Sep 13 16:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.67200000000000004,"15min":0.64300000000000002},"min":{"1min":0.46000000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83899999999999997,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.64500000000000002,"15min":0.61299999999999999},"min":{"1min":0.42399999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.85099999999999998,"5min":4.3170000000000002,"15min":4.3170000000000002},"last":0.78500000000000003}]},{"osd":3,"last update":"Fri Sep 13 16:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.91400000000000003,"15min":0.68899999999999995},"min":{"1min":0.46100000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.89500000000000002,"5min":28.707000000000001,"15min":28.707000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.68799999999999994,"15min":0.625},"min":{"1min":0.43099999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88300000000000001,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":0.77300000000000002}]},{"osd":4,"last update":"Fri Sep 13 16:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.40799999999999997,"15min":0.44700000000000001},"min":{"1min":0.29299999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.59899999999999998,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.44700000000000001,"15min":0.47799999999999998},"min":{"1min":0.34899999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.68799999999999994,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.44600000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.44,"15min":0.45600000000000002},"min":{"1min":0.32600000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.70799999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45700000000000002,"15min":0.48299999999999998},"min":{"1min":0.377,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.54800000000000004,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.42299999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.48299999999999998,"15min":0.51700000000000002},"min":{"1min":0.34499999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.504},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.498,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.46999999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574915,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39552,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94332288,"statfs":{"total":96636764160,"available":96596262912,"internally_reserved":0,"allocated":253952,"data_stored":84762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.58999999999999997,"15min":0.57599999999999996},"min":{"1min":0.47199999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.84799999999999998,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60999999999999999,"15min":0.59399999999999997},"min":{"1min":0.35899999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78300000000000003,"5min":2.9980000000000002,"15min":2.9980000000000002},"last":0.505}]},{"osd":1,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.70099999999999996,"15min":0.63100000000000001},"min":{"1min":0.40400000000000003,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.75800000000000001,"5min":4.3700000000000001,"15min":4.3700000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.63100000000000001,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82899999999999996,"5min":3.9020000000000001,"15min":3.9020000000000001},"last":0.55000000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59899999999999998,"15min":0.59999999999999998},"min":{"1min":0.46500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.77400000000000002,"5min":2.613,"15min":2.613},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.72799999999999998,"15min":0.65900000000000003},"min":{"1min":0.47399999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81100000000000005,"5min":7.9809999999999999,"15min":7.9809999999999999},"last":0.64500000000000002}]},{"osd":3,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66600000000000004,"15min":0.63600000000000001},"min":{"1min":0.45500000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.80700000000000005,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.67600000000000005,"15min":0.68600000000000005},"min":{"1min":0.49199999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81699999999999995,"5min":2.8130000000000002,"15min":2.8130000000000002},"last":0.65500000000000003}]},{"osd":5,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.41899999999999998,"15min":0.45800000000000002},"min":{"1min":0.315,"5min":0.249,"15min":0.249},"max":{"1min":0.61199999999999999,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.43099999999999999,"15min":0.47099999999999997},"min":{"1min":0.34999999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.42099999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.432,"15min":0.47699999999999998},"min":{"1min":0.35199999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67500000000000004,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46899999999999997,"15min":0.52800000000000002},"min":{"1min":0.33700000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57799999999999996,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.38}]},{"osd":7,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47299999999999998,"15min":0.50700000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64900000000000002,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.51700000000000002,"15min":0.59499999999999997},"min":{"1min":0.32100000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70099999999999996,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.40600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574914,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44012,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94327828,"statfs":{"total":96636764160,"available":96591695872,"internally_reserved":0,"allocated":626688,"data_stored":449059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":44431816},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.57899999999999996,"15min":0.49199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.64000000000000001,"5min":4.8449999999999998,"15min":4.8449999999999998},"last":0.432},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.78000000000000003,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.78700000000000003,"5min":7.7489999999999997,"15min":7.7489999999999997},"last":0.51900000000000002}]},{"osd":1,"last update":"Fri Sep 13 16:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.84199999999999997,"15min":0.60499999999999998},"min":{"1min":0.379,"5min":0.252,"15min":0.252},"max":{"1min":0.628,"5min":28.908999999999999,"15min":28.908999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.71599999999999997,"15min":0.64000000000000001},"min":{"1min":0.38,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.70899999999999996,"5min":3.1699999999999999,"15min":3.1699999999999999},"last":0.69599999999999995}]},{"osd":2,"last update":"Fri Sep 13 16:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.80900000000000005,"15min":0.63700000000000001},"min":{"1min":0.38800000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86699999999999999,"5min":4.3630000000000004,"15min":4.3630000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.75700000000000001,"15min":0.68899999999999995},"min":{"1min":0.44800000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.81799999999999995,"5min":3.2610000000000001,"15min":3.2610000000000001},"last":0.76500000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.94299999999999995,"15min":0.82699999999999996},"min":{"1min":0.624,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.98499999999999999,"5min":4.1029999999999998,"15min":4.1029999999999998},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.78900000000000003,"15min":0.752},"min":{"1min":0.55400000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89000000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":0.71799999999999997}]},{"osd":5,"last update":"Fri Sep 13 16:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.80200000000000005,"15min":0.752},"min":{"1min":0.54700000000000004,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.79800000000000004,"5min":2.5790000000000002,"15min":2.5790000000000002},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.871,"15min":0.84099999999999997},"min":{"1min":0.57999999999999996,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.89200000000000002,"5min":3.1970000000000001,"15min":3.1970000000000001},"last":0.81100000000000005}]},{"osd":6,"last update":"Fri Sep 13 16:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.96399999999999997,"15min":0.84099999999999997},"min":{"1min":0.56599999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.96799999999999997,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.85299999999999998,"15min":0.82399999999999995},"min":{"1min":0.58899999999999997,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.877,"5min":2.7360000000000002,"15min":2.7360000000000002},"last":0.78900000000000003}]},{"osd":7,"last update":"Fri Sep 13 16:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.873,"15min":0.81999999999999995},"min":{"1min":0.60399999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.97199999999999998,"5min":3.2320000000000002,"15min":3.2320000000000002},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":1.002,"15min":0.873},"min":{"1min":0.625,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.016,"5min":4.181,"15min":4.181},"last":0.83199999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35464,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336376,"statfs":{"total":96636764160,"available":96600449024,"internally_reserved":0,"allocated":262144,"data_stored":84385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.98599999999999999,"15min":0.65500000000000003},"min":{"1min":0.315,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.71199999999999997,"5min":31.440999999999999,"15min":31.440999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.68400000000000005,"15min":0.57399999999999995},"min":{"1min":0.27600000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.72399999999999998,"5min":4.7720000000000002,"15min":4.7720000000000002},"last":0.371}]},{"osd":1,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.89200000000000002,"15min":0.65400000000000003},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72499999999999998,"5min":8.7780000000000005,"15min":8.7780000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.752,"15min":0.61199999999999999},"min":{"1min":0.36899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.754,"5min":6.5659999999999998,"15min":6.5659999999999998},"last":0.54600000000000004}]},{"osd":3,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.70999999999999996,"15min":0.59199999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75900000000000001,"5min":6.5469999999999997,"15min":6.5469999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.84499999999999997,"15min":0.68300000000000005},"min":{"1min":0.45600000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79500000000000004,"5min":8.8659999999999997,"15min":8.8659999999999997},"last":0.56499999999999995}]},{"osd":4,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.81299999999999994,"15min":0.751},"min":{"1min":0.53800000000000003,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88900000000000001,"5min":3.4460000000000002,"15min":3.4460000000000002},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.79900000000000004,"15min":0.76100000000000001},"min":{"1min":0.51300000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90700000000000003,"5min":6.4379999999999997,"15min":6.4379999999999997},"last":0.70999999999999996}]},{"osd":5,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.97299999999999998,"15min":0.81599999999999995},"min":{"1min":0.60199999999999998,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93999999999999995,"5min":9.1039999999999992,"15min":9.1039999999999992},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.755,"5min":0.84699999999999998,"15min":0.77500000000000002},"min":{"1min":0.61799999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.92900000000000005,"5min":3.633,"15min":3.633},"last":0.68500000000000005}]},{"osd":6,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.82599999999999996,"15min":0.79200000000000004},"min":{"1min":0.57899999999999996,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.93899999999999995,"5min":6.5140000000000002,"15min":6.5140000000000002},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.99399999999999999,"15min":0.85799999999999998},"min":{"1min":0.60399999999999998,"5min":0.442,"15min":0.442},"max":{"1min":0.91300000000000003,"5min":9.141,"15min":9.141},"last":0.73899999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.86799999999999999,"15min":0.83799999999999997},"min":{"1min":0.53400000000000003,"5min":0.505,"15min":0.505},"max":{"1min":0.95599999999999996,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.85499999999999998,"15min":0.83099999999999996},"min":{"1min":0.58199999999999996,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.97199999999999998,"5min":6.4800000000000004,"15min":6.4800000000000004},"last":0.79500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35460,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336380,"statfs":{"total":96636764160,"available":96600453120,"internally_reserved":0,"allocated":258048,"data_stored":84564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.99099999999999999,"15min":0.64900000000000002},"min":{"1min":0.30599999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.64900000000000002,"5min":22.721,"15min":22.721},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.876,"15min":0.64400000000000002},"min":{"1min":0.34200000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.194,"5min":19.494,"15min":19.494},"last":0.55300000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.98599999999999999,"15min":0.67100000000000004},"min":{"1min":0.34300000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.93799999999999994,"5min":15.491,"15min":15.491},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.92500000000000004,"15min":0.70099999999999996},"min":{"1min":0.36899999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.5720000000000001,"5min":17.071000000000002,"15min":17.071000000000002},"last":0.68400000000000005}]},{"osd":3,"last update":"Fri Sep 13 16:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.98099999999999998,"15min":0.73299999999999998},"min":{"1min":0.40699999999999997,"5min":0.374,"15min":0.374},"max":{"1min":0.79700000000000004,"5min":18.838000000000001,"15min":18.838000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.91700000000000004,"15min":0.69499999999999995},"min":{"1min":0.33200000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.91500000000000004,"5min":20.199000000000002,"15min":20.199000000000002},"last":0.64200000000000002}]},{"osd":4,"last update":"Fri Sep 13 16:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.234,"15min":0.91200000000000003},"min":{"1min":0.45000000000000001,"5min":0.373,"15min":0.373},"max":{"1min":1.1080000000000001,"5min":15.654999999999999,"15min":15.654999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":1.085,"15min":0.89300000000000002},"min":{"1min":0.54600000000000004,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.536,"5min":18.742000000000001,"15min":18.742000000000001},"last":0.75600000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.95999999999999996,"15min":0.88700000000000001},"min":{"1min":0.57699999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.98699999999999999,"5min":7.3230000000000004,"15min":7.3230000000000004},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.751,"5min":1.2809999999999999,"15min":0.97799999999999998},"min":{"1min":0.52100000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.028,"5min":15.686,"15min":15.686},"last":0.81899999999999995}]},{"osd":6,"last update":"Fri Sep 13 16:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.96299999999999997,"15min":0.86499999999999999},"min":{"1min":0.57399999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.96399999999999997,"5min":7.3700000000000001,"15min":7.3700000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.76200000000000001,"5min":1.302,"15min":0.997},"min":{"1min":0.56399999999999995,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.069,"5min":15.718,"15min":15.718},"last":0.86299999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":1.1180000000000001,"15min":0.89400000000000002},"min":{"1min":0.51300000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.49,"5min":18.798999999999999,"15min":18.798999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.97599999999999998,"15min":0.89600000000000002},"min":{"1min":0.52800000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0169999999999999,"5min":7.4950000000000001,"15min":7.4950000000000001},"last":0.60099999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574914,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43712,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":43390,"kb_avail":94328128,"statfs":{"total":96636764160,"available":96592003072,"internally_reserved":0,"allocated":319488,"data_stored":673303,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":44431814},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.64400000000000002,"15min":0.52500000000000002},"min":{"1min":0.33400000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.61499999999999999,"5min":7.2539999999999996,"15min":7.2539999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.72999999999999998,"15min":0.56000000000000005},"min":{"1min":0.34799999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.68899999999999995,"5min":6.0949999999999998,"15min":6.0949999999999998},"last":0.51100000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.77300000000000002,"15min":0.623},"min":{"1min":0.38600000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70699999999999996,"5min":7.1920000000000002,"15min":7.1920000000000002},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.74299999999999999,"15min":0.59899999999999998},"min":{"1min":0.41099999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72499999999999998,"5min":6.5,"15min":6.5},"last":0.56399999999999995}]},{"osd":3,"last update":"Fri Sep 13 16:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.66900000000000004,"15min":0.54900000000000004},"min":{"1min":0.39900000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88500000000000001,"5min":4.4459999999999997,"15min":4.4459999999999997},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.85399999999999998,"15min":0.68899999999999995},"min":{"1min":0.43099999999999999,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.84999999999999998,"5min":10.522,"15min":10.522},"last":0.622}]},{"osd":4,"last update":"Fri Sep 13 16:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.88500000000000001,"15min":0.79800000000000004},"min":{"1min":0.51400000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92100000000000004,"5min":6.4560000000000004,"15min":6.4560000000000004},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.82299999999999995,"15min":0.77700000000000002},"min":{"1min":0.53200000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89500000000000002,"5min":3.79,"15min":3.79},"last":0.83899999999999997}]},{"osd":5,"last update":"Fri Sep 13 16:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.84799999999999998,"15min":0.78600000000000003},"min":{"1min":0.51900000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.024,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.94699999999999995,"15min":0.84699999999999998},"min":{"1min":0.54900000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.92300000000000004,"5min":7.8339999999999996,"15min":7.8339999999999996},"last":0.81399999999999995}]},{"osd":6,"last update":"Fri Sep 13 16:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.97299999999999998,"15min":0.86199999999999999},"min":{"1min":0.59699999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.95199999999999996,"5min":7.5510000000000002,"15min":7.5510000000000002},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.95599999999999996,"15min":0.81499999999999995},"min":{"1min":0.56000000000000005,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.96199999999999997,"5min":7.444,"15min":7.444},"last":0.79700000000000004}]},{"osd":7,"last update":"Fri Sep 13 16:48:59 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.95399999999999996,"15min":0.81599999999999995},"min":{"1min":0.51900000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.93899999999999995,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.879,"15min":0.79900000000000004},"min":{"1min":0.625,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.103,"5min":4.1879999999999997,"15min":4.1879999999999997},"last":0.88800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32543,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":365979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":389120,"data_stored":365840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:49:11.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-13T16:49:11.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.397+0000 7ff74ec00640 1 Processor -- start 2024-09-13T16:49:11.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.397+0000 7ff74ec00640 1 -- start start 2024-09-13T16:49:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.398+0000 7ff74ec00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff74815a440 0x7ff7480c5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.398+0000 7ff74ec00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff74806dcb0 0x7ff7481504e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.399+0000 7ff74ec00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7480c1d80 con 0x7ff74815a060 2024-09-13T16:49:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.399+0000 7ff74ec00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff74815a920 con 0x7ff74815a440 2024-09-13T16:49:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.399+0000 7ff74ec00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff74815ac00 con 0x7ff74806dcb0 2024-09-13T16:49:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.399+0000 7ff73fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff74815a440 0x7ff7480c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.399+0000 7ff74cc00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff74806dcb0 0x7ff7481504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.399+0000 7ff73fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff74815a440 0x7ff7480c5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35640/0 (socket says 172.21.15.28:35640) 2024-09-13T16:49:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.399+0000 7ff747e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff74815a060 0x7ff74806d5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55248/0 (socket says 172.21.15.28:55248) 2024-09-13T16:49:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.399+0000 7ff747e00640 1 -- 172.21.15.28:0/2310667769 learned_addr learned my addr 172.21.15.28:0/2310667769 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:49:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.399+0000 7ff74cc00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff74806dcb0 0x7ff7481504e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.28:46392/0 (socket says 172.21.15.28:46392) 2024-09-13T16:49:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.399+0000 7ff73fe00640 1 -- 172.21.15.28:0/2310667769 >> v2:172.21.15.191:3300/0 conn(0x7ff74806dcb0 msgr2=0x7ff7481504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:49:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.399+0000 7ff73fe00640 1 --2- 172.21.15.28:0/2310667769 >> v2:172.21.15.191:3300/0 conn(0x7ff74806dcb0 0x7ff7481504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.399+0000 7ff73fe00640 1 -- 172.21.15.28:0/2310667769 >> v1:172.21.15.28:6789/0 conn(0x7ff74815a060 legacy=0x7ff74806d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.399+0000 7ff73fe00640 1 -- 172.21.15.28:0/2310667769 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7480c2d50 con 0x7ff74815a440 2024-09-13T16:49:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.399+0000 7ff747400640 1 -- 172.21.15.28:0/2310667769 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706162639 0 0) 0x7ff7480c1d80 con 0x7ff74815a060 2024-09-13T16:49:11.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.400+0000 7ff73fe00640 1 --2- 172.21.15.28:0/2310667769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff74815a440 0x7ff7480c5500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff7480c20b0 tx=0x7ff73002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a84bb0ab616260a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.401+0000 7ff747400640 1 -- 172.21.15.28:0/2310667769 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff730032440 con 0x7ff74815a440 2024-09-13T16:49:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.401+0000 7ff747400640 1 -- 172.21.15.28:0/2310667769 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7300325a0 con 0x7ff74815a440 2024-09-13T16:49:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.401+0000 7ff747400640 1 -- 172.21.15.28:0/2310667769 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff730032930 con 0x7ff74815a440 2024-09-13T16:49:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.401+0000 7ff74ec00640 1 -- 172.21.15.28:0/2310667769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff74815a440 msgr2=0x7ff7480c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.401+0000 7ff74ec00640 1 --2- 172.21.15.28:0/2310667769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff74815a440 0x7ff7480c5500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff7480c20b0 tx=0x7ff73002a1a0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.401+0000 7ff74ec00640 1 -- 172.21.15.28:0/2310667769 shutdown_connections 2024-09-13T16:49:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.401+0000 7ff74ec00640 1 --2- 172.21.15.28:0/2310667769 >> v2:172.21.15.191:3300/0 conn(0x7ff74806dcb0 0x7ff7481504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.401+0000 7ff74ec00640 1 --2- 172.21.15.28:0/2310667769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff74815a440 0x7ff7480c5500 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.401+0000 7ff74ec00640 1 -- 172.21.15.28:0/2310667769 >> 172.21.15.28:0/2310667769 conn(0x7ff7480ca6b0 msgr2=0x7ff7480caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.401+0000 7ff74ec00640 1 -- 172.21.15.28:0/2310667769 shutdown_connections 2024-09-13T16:49:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.401+0000 7ff74ec00640 1 -- 172.21.15.28:0/2310667769 wait complete. 2024-09-13T16:49:11.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.403+0000 7ff74ec00640 1 Processor -- start 2024-09-13T16:49:11.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.403+0000 7ff74ec00640 1 -- start start 2024-09-13T16:49:11.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.404+0000 7ff74ec00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff74806dcb0 0x7ff7481548e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:11.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.404+0000 7ff74ec00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff74815a060 0x7ff748158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.404+0000 7ff74ec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff74815a440 0x7ff7481592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.404+0000 7ff74ec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff748155500 con 0x7ff74815a440 2024-09-13T16:49:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.404+0000 7ff74ec00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7480c13f0 con 0x7ff74806dcb0 2024-09-13T16:49:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.404+0000 7ff74ec00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7481567b0 con 0x7ff74815a060 2024-09-13T16:49:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.404+0000 7ff74cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff74815a440 0x7ff7481592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.404+0000 7ff74cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff74815a440 0x7ff7481592f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35646/0 (socket says 172.21.15.28:35646) 2024-09-13T16:49:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.404+0000 7ff74cc00640 1 -- 172.21.15.28:0/3847617151 learned_addr learned my addr 172.21.15.28:0/3847617151 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:49:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.404+0000 7ff747e00640 1 --2- 172.21.15.28:0/3847617151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff74806dcb0 0x7ff7481548e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.404+0000 7ff73fe00640 1 --2- 172.21.15.28:0/3847617151 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff74815a060 0x7ff748158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.405+0000 7ff74cc00640 1 -- 172.21.15.28:0/3847617151 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff74815a060 msgr2=0x7ff748158db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.405+0000 7ff74cc00640 1 --2- 172.21.15.28:0/3847617151 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff74815a060 0x7ff748158db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.405+0000 7ff74cc00640 1 -- 172.21.15.28:0/3847617151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff74806dcb0 msgr2=0x7ff7481548e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.405+0000 7ff74cc00640 1 --2- 172.21.15.28:0/3847617151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff74806dcb0 0x7ff7481548e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.405+0000 7ff74cc00640 1 -- 172.21.15.28:0/3847617151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff730009400 con 0x7ff74815a440 2024-09-13T16:49:11.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.405+0000 7ff74cc00640 1 --2- 172.21.15.28:0/3847617151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff74815a440 0x7ff7481592f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff74000efc0 tx=0x7ff74000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:11.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.405+0000 7ff744c00640 1 -- 172.21.15.28:0/3847617151 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff740009ba0 con 0x7ff74815a440 2024-09-13T16:49:11.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.405+0000 7ff744c00640 1 -- 172.21.15.28:0/3847617151 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74000f040 con 0x7ff74815a440 2024-09-13T16:49:11.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.405+0000 7ff744c00640 1 -- 172.21.15.28:0/3847617151 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7400146c0 con 0x7ff74815a440 2024-09-13T16:49:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.405+0000 7ff74ec00640 1 -- 172.21.15.28:0/3847617151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff748157ff0 con 0x7ff74815a440 2024-09-13T16:49:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.406+0000 7ff74ec00640 1 -- 172.21.15.28:0/3847617151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7481585b0 con 0x7ff74815a440 2024-09-13T16:49:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.407+0000 7ff74ec00640 1 -- 172.21.15.28:0/3847617151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7480c62a0 con 0x7ff74815a440 2024-09-13T16:49:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.407+0000 7ff744c00640 1 -- 172.21.15.28:0/3847617151 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff740014820 con 0x7ff74815a440 2024-09-13T16:49:11.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.408+0000 7ff744c00640 1 --2- 172.21.15.28:0/3847617151 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff72406ac30 0x7ff72406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.408+0000 7ff744c00640 1 -- 172.21.15.28:0/3847617151 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff740089db0 con 0x7ff74815a440 2024-09-13T16:49:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.408+0000 7ff747e00640 1 --2- 172.21.15.28:0/3847617151 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff72406ac30 0x7ff72406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.409+0000 7ff747e00640 1 --2- 172.21.15.28:0/3847617151 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff72406ac30 0x7ff72406d0f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff738000f60 tx=0x7ff738007d50 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:11.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.412+0000 7ff744c00640 1 -- 172.21.15.28:0/3847617151 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff74008e070 con 0x7ff74815a440 2024-09-13T16:49:11.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.591+0000 7ff74ec00640 1 -- 172.21.15.28:0/3847617151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7ff748154220 con 0x7ff74815a440 2024-09-13T16:49:11.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.595+0000 7ff744c00640 1 -- 172.21.15.28:0/3847617151 <== mon.0 v2:172.21.15.28:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7ff740054fb0 con 0x7ff74815a440 2024-09-13T16:49:11.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.595+0000 7ff744c00640 1 -- 172.21.15.28:0/3847617151 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) v1 ==== 123+0+0 (secure 0 0 0) 0x7ff740018020 con 0x7ff74815a440 2024-09-13T16:49:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.598+0000 7ff74ec00640 1 -- 172.21.15.28:0/3847617151 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff72406ac30 msgr2=0x7ff72406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.598+0000 7ff74ec00640 1 --2- 172.21.15.28:0/3847617151 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff72406ac30 0x7ff72406d0f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff738000f60 tx=0x7ff738007d50 comp rx=0 tx=0).stop 2024-09-13T16:49:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.598+0000 7ff74ec00640 1 -- 172.21.15.28:0/3847617151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff74815a440 msgr2=0x7ff7481592f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.598+0000 7ff74ec00640 1 --2- 172.21.15.28:0/3847617151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff74815a440 0x7ff7481592f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff74000efc0 tx=0x7ff74000c490 comp rx=0 tx=0).stop 2024-09-13T16:49:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.598+0000 7ff74ec00640 1 -- 172.21.15.28:0/3847617151 shutdown_connections 2024-09-13T16:49:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.598+0000 7ff74ec00640 1 --2- 172.21.15.28:0/3847617151 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7ff72406ac30 0x7ff72406d0f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.598+0000 7ff74ec00640 1 --2- 172.21.15.28:0/3847617151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff74815a440 0x7ff7481592f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.598+0000 7ff74ec00640 1 --2- 172.21.15.28:0/3847617151 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff74815a060 0x7ff748158db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.598+0000 7ff74ec00640 1 --2- 172.21.15.28:0/3847617151 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff74806dcb0 0x7ff7481548e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.598+0000 7ff74ec00640 1 -- 172.21.15.28:0/3847617151 >> 172.21.15.28:0/3847617151 conn(0x7ff7480ca6b0 msgr2=0x7ff74814eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.598+0000 7ff74ec00640 1 -- 172.21.15.28:0/3847617151 shutdown_connections 2024-09-13T16:49:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.598+0000 7ff74ec00640 1 -- 172.21.15.28:0/3847617151 wait complete. 2024-09-13T16:49:11.616 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T16:49:11.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:49:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.774+0000 7f657d000640 1 Processor -- start 2024-09-13T16:49:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.775+0000 7f657d000640 1 -- start start 2024-09-13T16:49:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.776+0000 7f657d000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f657814f790 0x7f65780cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.776+0000 7f657d000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65780c42f0 0x7f6578070640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.776+0000 7f657d000640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657806cb60 con 0x7f657814fb70 2024-09-13T16:49:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.776+0000 7f657d000640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65780c1c80 con 0x7f657814f790 2024-09-13T16:49:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.776+0000 7f657d000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657806ce40 con 0x7f65780c42f0 2024-09-13T16:49:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.776+0000 7f6577e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65780c42f0 0x7f6578070640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.776+0000 7f6577400640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f657814f790 0x7f65780cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.776+0000 7f6577e00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65780c42f0 0x7f6578070640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.28:59182/0 (socket says 172.21.15.28:59182) 2024-09-13T16:49:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.776+0000 7f656fe00640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f657814fb70 0x7f65780c3140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:41008/0 (socket says 172.21.15.28:41008) 2024-09-13T16:49:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.776+0000 7f6577e00640 1 -- 172.21.15.28:0/507265381 learned_addr learned my addr 172.21.15.28:0/507265381 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:49:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.777+0000 7f6576a00640 1 -- 172.21.15.28:0/507265381 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424238388 0 0) 0x7f657806cb60 con 0x7f657814fb70 2024-09-13T16:49:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.777+0000 7f6576a00640 1 -- 172.21.15.28:0/507265381 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f655c003520 con 0x7f657814fb70 2024-09-13T16:49:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.777+0000 7f6577e00640 1 -- 172.21.15.28:0/507265381 >> v2:172.21.15.191:3300/0 conn(0x7f657814f790 msgr2=0x7f65780cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:49:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.777+0000 7f6577e00640 1 --2- 172.21.15.28:0/507265381 >> v2:172.21.15.191:3300/0 conn(0x7f657814f790 0x7f65780cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.777+0000 7f6577e00640 1 -- 172.21.15.28:0/507265381 >> v1:172.21.15.191:6789/0 conn(0x7f657814fb70 legacy=0x7f65780c3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.777+0000 7f6577e00640 1 -- 172.21.15.28:0/507265381 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6578156070 con 0x7f65780c42f0 2024-09-13T16:49:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.777+0000 7f6577400640 1 --2- 172.21.15.28:0/507265381 >> v2:172.21.15.191:3300/0 conn(0x7f657814f790 0x7f65780cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:49:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.778+0000 7f6577e00640 1 --2- 172.21.15.28:0/507265381 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65780c42f0 0x7f6578070640 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6570009e50 tx=0x7f657002a7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91ec53479cf00a64 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.778+0000 7f6576a00640 1 -- 172.21.15.28:0/507265381 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570031600 con 0x7f65780c42f0 2024-09-13T16:49:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.778+0000 7f6576a00640 1 -- 172.21.15.28:0/507265381 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f657002d050 con 0x7f65780c42f0 2024-09-13T16:49:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.778+0000 7f6576a00640 1 -- 172.21.15.28:0/507265381 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570031960 con 0x7f65780c42f0 2024-09-13T16:49:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.779+0000 7f657d000640 1 -- 172.21.15.28:0/507265381 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65780c42f0 msgr2=0x7f6578070640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.779+0000 7f657d000640 1 --2- 172.21.15.28:0/507265381 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65780c42f0 0x7f6578070640 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6570009e50 tx=0x7f657002a7a0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.779+0000 7f657d000640 1 -- 172.21.15.28:0/507265381 shutdown_connections 2024-09-13T16:49:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.779+0000 7f657d000640 1 --2- 172.21.15.28:0/507265381 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65780c42f0 0x7f6578070640 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.779+0000 7f657d000640 1 --2- 172.21.15.28:0/507265381 >> v2:172.21.15.191:3300/0 conn(0x7f657814f790 0x7f65780cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.779+0000 7f657d000640 1 -- 172.21.15.28:0/507265381 >> 172.21.15.28:0/507265381 conn(0x7f65780ca0e0 msgr2=0x7f65780ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.779+0000 7f657d000640 1 -- 172.21.15.28:0/507265381 shutdown_connections 2024-09-13T16:49:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.779+0000 7f657d000640 1 -- 172.21.15.28:0/507265381 wait complete. 2024-09-13T16:49:11.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.780+0000 7f657d000640 1 Processor -- start 2024-09-13T16:49:11.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.780+0000 7f657d000640 1 -- start start 2024-09-13T16:49:11.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.781+0000 7f657d000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65780c42f0 0x7f657806de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:11.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.781+0000 7f657d000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f657814f790 0x7f65781513d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:11.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.781+0000 7f657d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f657814e160 0x7f6578151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:11.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.781+0000 7f657d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65781ec390 con 0x7f657814e160 2024-09-13T16:49:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.781+0000 7f657d000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6578150cd0 con 0x7f657814f790 2024-09-13T16:49:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.781+0000 7f657d000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65781ec610 con 0x7f65780c42f0 2024-09-13T16:49:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.782+0000 7f6577e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f657814e160 0x7f6578151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.782+0000 7f656fe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f657814f790 0x7f65781513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.782+0000 7f6577e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f657814e160 0x7f6578151910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35652/0 (socket says 172.21.15.28:35652) 2024-09-13T16:49:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.782+0000 7f6577e00640 1 -- 172.21.15.28:0/1773868962 learned_addr learned my addr 172.21.15.28:0/1773868962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:49:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.782+0000 7f6577400640 1 --2- 172.21.15.28:0/1773868962 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65780c42f0 0x7f657806de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.782+0000 7f656fe00640 1 -- 172.21.15.28:0/1773868962 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65780c42f0 msgr2=0x7f657806de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.782+0000 7f656fe00640 1 --2- 172.21.15.28:0/1773868962 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65780c42f0 0x7f657806de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.782+0000 7f656fe00640 1 -- 172.21.15.28:0/1773868962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f657814e160 msgr2=0x7f6578151910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.782+0000 7f656fe00640 1 --2- 172.21.15.28:0/1773868962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f657814e160 0x7f6578151910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.782+0000 7f656fe00640 1 -- 172.21.15.28:0/1773868962 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6568009400 con 0x7f657814f790 2024-09-13T16:49:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.782+0000 7f656fe00640 1 --2- 172.21.15.28:0/1773868962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f657814f790 0x7f65781513d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f656000c390 tx=0x7f6560001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.783+0000 7f656f400640 1 -- 172.21.15.28:0/1773868962 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65600074e0 con 0x7f657814f790 2024-09-13T16:49:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.783+0000 7f656f400640 1 -- 172.21.15.28:0/1773868962 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f6560007640 con 0x7f657814f790 2024-09-13T16:49:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.783+0000 7f656f400640 1 -- 172.21.15.28:0/1773868962 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6560004a20 con 0x7f657814f790 2024-09-13T16:49:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.783+0000 7f657d000640 1 -- 172.21.15.28:0/1773868962 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6570009b30 con 0x7f657814f790 2024-09-13T16:49:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.783+0000 7f657d000640 1 -- 172.21.15.28:0/1773868962 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65781ecbd0 con 0x7f657814f790 2024-09-13T16:49:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.784+0000 7f657d000640 1 -- 172.21.15.28:0/1773868962 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6578066f40 con 0x7f657814f790 2024-09-13T16:49:11.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.785+0000 7f656f400640 1 -- 172.21.15.28:0/1773868962 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6560004b80 con 0x7f657814f790 2024-09-13T16:49:11.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.786+0000 7f656f400640 1 --2- 172.21.15.28:0/1773868962 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f655406ac30 0x7f655406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:11.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.786+0000 7f656f400640 1 -- 172.21.15.28:0/1773868962 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6560088cb0 con 0x7f657814f790 2024-09-13T16:49:11.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.786+0000 7f6577400640 1 --2- 172.21.15.28:0/1773868962 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f655406ac30 0x7f655406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:11.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.786+0000 7f6577400640 1 --2- 172.21.15.28:0/1773868962 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f655406ac30 0x7f655406d0f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6568009cd0 tx=0x7f6568012040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:11.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:11.790+0000 7f656f400640 1 -- 172.21.15.28:0/1773868962 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f65600578b0 con 0x7f657814f790 2024-09-13T16:49:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.005+0000 7f657d000640 1 -- 172.21.15.28:0/1773868962 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f657806ae70 con 0x7f657814f790 2024-09-13T16:49:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.006+0000 7f656f400640 1 -- 172.21.15.28:0/1773868962 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6307 (secure 0 0 0) 0x7f6560057440 con 0x7f657814f790 2024-09-13T16:49:12.008 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:49:12.008 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4227,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/355570369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":355570369},{"type":"v1","addr":"172.21.15.28:6835","nonce":355570369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:43:54.969699+0000","modified":"2024-09-13T16:44:26.685528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4250,"mds_2":4244,"mds_3":4235,"mds_4":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"f","rank":4,"incarnation":16,"state":"up:active","state_seq":10,"addr":"172.21.15.191:6839/2366383263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6837","nonce":2366383263},{"type":"v1","addr":"172.21.15.191:6839","nonce":2366383263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4235":{"gid":4235,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.191:6838/2702860767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2702860767},{"type":"v1","addr":"172.21.15.191:6838","nonce":2702860767}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/2240589617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2240589617},{"type":"v1","addr":"172.21.15.28:6833","nonce":2240589617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6837/4068563017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":4068563017},{"type":"v1","addr":"172.21.15.28:6837","nonce":4068563017}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4258":{"gid":4258,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/704943585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":704943585},{"type":"v1","addr":"172.21.15.191:6835","nonce":704943585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T16:49:12.008 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-09-13T16:49:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.008+0000 7f657d000640 1 -- 172.21.15.28:0/1773868962 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f655406ac30 msgr2=0x7f655406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.008+0000 7f657d000640 1 --2- 172.21.15.28:0/1773868962 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f655406ac30 0x7f655406d0f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6568009cd0 tx=0x7f6568012040 comp rx=0 tx=0).stop 2024-09-13T16:49:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.008+0000 7f657d000640 1 -- 172.21.15.28:0/1773868962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f657814f790 msgr2=0x7f65781513d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.008+0000 7f657d000640 1 --2- 172.21.15.28:0/1773868962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f657814f790 0x7f65781513d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f656000c390 tx=0x7f6560001e70 comp rx=0 tx=0).stop 2024-09-13T16:49:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.008+0000 7f657d000640 1 -- 172.21.15.28:0/1773868962 shutdown_connections 2024-09-13T16:49:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.008+0000 7f657d000640 1 --2- 172.21.15.28:0/1773868962 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f655406ac30 0x7f655406d0f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.008+0000 7f657d000640 1 --2- 172.21.15.28:0/1773868962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f657814e160 0x7f6578151910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.008+0000 7f657d000640 1 --2- 172.21.15.28:0/1773868962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f657814f790 0x7f65781513d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.008+0000 7f657d000640 1 --2- 172.21.15.28:0/1773868962 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65780c42f0 0x7f657806de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.008+0000 7f657d000640 1 -- 172.21.15.28:0/1773868962 >> 172.21.15.28:0/1773868962 conn(0x7f65780ca0e0 msgr2=0x7f65780c9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.009+0000 7f657d000640 1 -- 172.21.15.28:0/1773868962 shutdown_connections 2024-09-13T16:49:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.009+0000 7f657d000640 1 -- 172.21.15.28:0/1773868962 wait complete. 2024-09-13T16:49:12.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:49:12.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.183+0000 7f93ffe00640 1 Processor -- start 2024-09-13T16:49:12.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.183+0000 7f93ffe00640 1 -- start start 2024-09-13T16:49:12.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.184+0000 7f93ffe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f93f80c3670 0x7f93f80c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.185+0000 7f93ffe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f814ca00 con 0x7f93f80c5bd0 2024-09-13T16:49:12.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.185+0000 7f93ffe00640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f8155990 con 0x7f93f80c1ea0 2024-09-13T16:49:12.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.185+0000 7f93ffe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93f80c2dc0 con 0x7f93f80c3670 2024-09-13T16:49:12.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.185+0000 7f93fca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f93f80c3670 0x7f93f80c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.185+0000 7f93fca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f93f80c3670 0x7f93f80c1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35668/0 (socket says 172.21.15.28:35668) 2024-09-13T16:49:12.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.185+0000 7f93fd400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f93f80c5bd0 0x7f93f8156150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55250/0 (socket says 172.21.15.28:55250) 2024-09-13T16:49:12.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.185+0000 7f93fca00640 1 -- 172.21.15.28:0/3448929425 learned_addr learned my addr 172.21.15.28:0/3448929425 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:49:12.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.185+0000 7f93fca00640 1 -- 172.21.15.28:0/3448929425 >> v1:172.21.15.191:6789/0 conn(0x7f93f80c1ea0 legacy=0x7f93f80c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.185+0000 7f93fca00640 1 -- 172.21.15.28:0/3448929425 >> v1:172.21.15.28:6789/0 conn(0x7f93f80c5bd0 legacy=0x7f93f8156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.185+0000 7f93f7e00640 1 -- 172.21.15.28:0/3448929425 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127976218 0 0) 0x7f93f8155990 con 0x7f93f80c1ea0 2024-09-13T16:49:12.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.185+0000 7f93fca00640 1 -- 172.21.15.28:0/3448929425 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f80c3b60 con 0x7f93f80c3670 2024-09-13T16:49:12.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.186+0000 7f93fca00640 1 --2- 172.21.15.28:0/3448929425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f80c3670 0x7f93f80c1960 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f93e400bb10 tx=0x7f93e402c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed15096c4c16192 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:12.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.187+0000 7f93f7e00640 1 -- 172.21.15.28:0/3448929425 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e402f070 con 0x7f93f80c3670 2024-09-13T16:49:12.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.187+0000 7f93f7e00640 1 -- 172.21.15.28:0/3448929425 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f93e40325a0 con 0x7f93f80c3670 2024-09-13T16:49:12.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.187+0000 7f93f7e00640 1 -- 172.21.15.28:0/3448929425 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e4032950 con 0x7f93f80c3670 2024-09-13T16:49:12.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.187+0000 7f93ffe00640 1 -- 172.21.15.28:0/3448929425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f80c3670 msgr2=0x7f93f80c1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.187+0000 7f93ffe00640 1 --2- 172.21.15.28:0/3448929425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f80c3670 0x7f93f80c1960 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f93e400bb10 tx=0x7f93e402c960 comp rx=0 tx=0).stop 2024-09-13T16:49:12.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.187+0000 7f93ffe00640 1 -- 172.21.15.28:0/3448929425 shutdown_connections 2024-09-13T16:49:12.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.187+0000 7f93ffe00640 1 --2- 172.21.15.28:0/3448929425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f80c3670 0x7f93f80c1960 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.187+0000 7f93ffe00640 1 -- 172.21.15.28:0/3448929425 >> 172.21.15.28:0/3448929425 conn(0x7f93f8066040 msgr2=0x7f93f8069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:12.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.187+0000 7f93ffe00640 1 -- 172.21.15.28:0/3448929425 shutdown_connections 2024-09-13T16:49:12.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.187+0000 7f93ffe00640 1 -- 172.21.15.28:0/3448929425 wait complete. 2024-09-13T16:49:12.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.189+0000 7f93ffe00640 1 Processor -- start 2024-09-13T16:49:12.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.189+0000 7f93ffe00640 1 -- start start 2024-09-13T16:49:12.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.190+0000 7f93ffe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f93f80c1ea0 0x7f93f814a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.190+0000 7f93ffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f80c3670 0x7f93f81e0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.190+0000 7f93ffe00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93f80c5bd0 0x7f93f81eb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.190+0000 7f93ffe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f81e0050 con 0x7f93f80c3670 2024-09-13T16:49:12.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.190+0000 7f93ffe00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f81e1740 con 0x7f93f80c5bd0 2024-09-13T16:49:12.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.190+0000 7f93ffe00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f81e1c70 con 0x7f93f80c1ea0 2024-09-13T16:49:12.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.190+0000 7f93fca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f80c3670 0x7f93f81e0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.190+0000 7f93fca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f80c3670 0x7f93f81e0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35680/0 (socket says 172.21.15.28:35680) 2024-09-13T16:49:12.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.190+0000 7f93fca00640 1 -- 172.21.15.28:0/2348052493 learned_addr learned my addr 172.21.15.28:0/2348052493 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:49:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.190+0000 7f93fde00640 1 --2- 172.21.15.28:0/2348052493 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93f80c5bd0 0x7f93f81eb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.190+0000 7f93fd400640 1 --2- 172.21.15.28:0/2348052493 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f93f80c1ea0 0x7f93f814a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.191+0000 7f93fca00640 1 -- 172.21.15.28:0/2348052493 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f93f80c1ea0 msgr2=0x7f93f814a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.191+0000 7f93fca00640 1 --2- 172.21.15.28:0/2348052493 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f93f80c1ea0 0x7f93f814a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.191+0000 7f93fca00640 1 -- 172.21.15.28:0/2348052493 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93f80c5bd0 msgr2=0x7f93f81eb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.191+0000 7f93fca00640 1 --2- 172.21.15.28:0/2348052493 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93f80c5bd0 0x7f93f81eb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.191+0000 7f93fca00640 1 -- 172.21.15.28:0/2348052493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e400b770 con 0x7f93f80c3670 2024-09-13T16:49:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.191+0000 7f93fca00640 1 --2- 172.21.15.28:0/2348052493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f80c3670 0x7f93f81e0cf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f93e400b2c0 tx=0x7f93e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.191+0000 7f93f5600640 1 -- 172.21.15.28:0/2348052493 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e4033770 con 0x7f93f80c3670 2024-09-13T16:49:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.191+0000 7f93f5600640 1 -- 172.21.15.28:0/2348052493 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f93e40338d0 con 0x7f93f80c3670 2024-09-13T16:49:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.191+0000 7f93f5600640 1 -- 172.21.15.28:0/2348052493 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e4033ca0 con 0x7f93f80c3670 2024-09-13T16:49:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.191+0000 7f93ffe00640 1 -- 172.21.15.28:0/2348052493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f81df6f0 con 0x7f93f80c3670 2024-09-13T16:49:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.191+0000 7f93ffe00640 1 -- 172.21.15.28:0/2348052493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f81df990 con 0x7f93f80c3670 2024-09-13T16:49:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.192+0000 7f93ffe00640 1 -- 172.21.15.28:0/2348052493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f8070680 con 0x7f93f80c3670 2024-09-13T16:49:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.193+0000 7f93f5600640 1 -- 172.21.15.28:0/2348052493 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f93e4044070 con 0x7f93f80c3670 2024-09-13T16:49:12.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.194+0000 7f93f5600640 1 --2- 172.21.15.28:0/2348052493 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f93d806ac30 0x7f93d806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.194+0000 7f93f5600640 1 -- 172.21.15.28:0/2348052493 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f93e40abfb0 con 0x7f93f80c3670 2024-09-13T16:49:12.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.194+0000 7f93fd400640 1 --2- 172.21.15.28:0/2348052493 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f93d806ac30 0x7f93d806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.195+0000 7f93fd400640 1 --2- 172.21.15.28:0/2348052493 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f93d806ac30 0x7f93d806d0f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f93ec000f60 tx=0x7f93ec007e20 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:12.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.198+0000 7f93f5600640 1 -- 172.21.15.28:0/2348052493 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f93e4077120 con 0x7f93f80c3670 2024-09-13T16:49:12.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.413+0000 7f93ffe00640 1 -- 172.21.15.28:0/2348052493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93f806a020 con 0x7f93f80c3670 2024-09-13T16:49:12.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.413+0000 7f93f5600640 1 -- 172.21.15.28:0/2348052493 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6307 (secure 0 0 0) 0x7f93e404b020 con 0x7f93f80c3670 2024-09-13T16:49:12.415 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:49:12.415 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4227,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/355570369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":355570369},{"type":"v1","addr":"172.21.15.28:6835","nonce":355570369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:43:54.969699+0000","modified":"2024-09-13T16:44:26.685528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4250,"mds_2":4244,"mds_3":4235,"mds_4":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"f","rank":4,"incarnation":16,"state":"up:active","state_seq":10,"addr":"172.21.15.191:6839/2366383263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6837","nonce":2366383263},{"type":"v1","addr":"172.21.15.191:6839","nonce":2366383263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4235":{"gid":4235,"name":"b","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.191:6838/2702860767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2702860767},{"type":"v1","addr":"172.21.15.191:6838","nonce":2702860767}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4244":{"gid":4244,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/2240589617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2240589617},{"type":"v1","addr":"172.21.15.28:6833","nonce":2240589617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4250":{"gid":4250,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6837/4068563017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":4068563017},{"type":"v1","addr":"172.21.15.28:6837","nonce":4068563017}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4258":{"gid":4258,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/704943585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":704943585},{"type":"v1","addr":"172.21.15.191:6835","nonce":704943585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T16:49:12.415 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-09-13T16:49:12.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.416+0000 7f93ffe00640 1 -- 172.21.15.28:0/2348052493 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f93d806ac30 msgr2=0x7f93d806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.416+0000 7f93ffe00640 1 --2- 172.21.15.28:0/2348052493 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f93d806ac30 0x7f93d806d0f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f93ec000f60 tx=0x7f93ec007e20 comp rx=0 tx=0).stop 2024-09-13T16:49:12.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.416+0000 7f93ffe00640 1 -- 172.21.15.28:0/2348052493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f80c3670 msgr2=0x7f93f81e0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.416+0000 7f93ffe00640 1 --2- 172.21.15.28:0/2348052493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f80c3670 0x7f93f81e0cf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f93e400b2c0 tx=0x7f93e40046c0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.416+0000 7f93ffe00640 1 -- 172.21.15.28:0/2348052493 shutdown_connections 2024-09-13T16:49:12.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.416+0000 7f93ffe00640 1 --2- 172.21.15.28:0/2348052493 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f93d806ac30 0x7f93d806d0f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.416+0000 7f93ffe00640 1 --2- 172.21.15.28:0/2348052493 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f93f80c5bd0 0x7f93f81eb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.416+0000 7f93ffe00640 1 --2- 172.21.15.28:0/2348052493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f80c3670 0x7f93f81e0cf0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.416+0000 7f93ffe00640 1 --2- 172.21.15.28:0/2348052493 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f93f80c1ea0 0x7f93f814a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.416+0000 7f93ffe00640 1 -- 172.21.15.28:0/2348052493 >> 172.21.15.28:0/2348052493 conn(0x7f93f8066040 msgr2=0x7f93f806e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:12.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.416+0000 7f93ffe00640 1 -- 172.21.15.28:0/2348052493 shutdown_connections 2024-09-13T16:49:12.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.416+0000 7f93ffe00640 1 -- 172.21.15.28:0/2348052493 wait complete. 2024-09-13T16:49:12.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:49:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.590+0000 7fc09da00640 1 Processor -- start 2024-09-13T16:49:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.590+0000 7fc09da00640 1 -- start start 2024-09-13T16:49:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.591+0000 7fc09da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc09814f790 0x7fc0980cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.591+0000 7fc09da00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc09814fb70 0x7fc0980c3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.591+0000 7fc09da00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc0980c42f0 0x7fc0980c3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.591+0000 7fc09da00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0980c1c80 con 0x7fc09814f790 2024-09-13T16:49:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.591+0000 7fc09da00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc09806cb60 con 0x7fc0980c42f0 2024-09-13T16:49:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.591+0000 7fc09da00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09806ce40 con 0x7fc09814fb70 2024-09-13T16:49:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.591+0000 7fc097e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc09814f790 0x7fc0980cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.592+0000 7fc097e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc09814f790 0x7fc0980cabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35684/0 (socket says 172.21.15.28:35684) 2024-09-13T16:49:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.592+0000 7fc097e00640 1 -- 172.21.15.28:0/2662993091 learned_addr learned my addr 172.21.15.28:0/2662993091 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:49:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.592+0000 7fc09d000640 1 --2- 172.21.15.28:0/2662993091 >> v2:172.21.15.191:3300/0 conn(0x7fc0980c42f0 0x7fc0980c3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.592+0000 7fc097400640 1 --2- 172.21.15.28:0/2662993091 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc09814fb70 0x7fc0980c3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.592+0000 7fc097e00640 1 -- 172.21.15.28:0/2662993091 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc09814fb70 msgr2=0x7fc0980c3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:49:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.592+0000 7fc097e00640 1 --2- 172.21.15.28:0/2662993091 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc09814fb70 0x7fc0980c3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.592+0000 7fc097e00640 1 -- 172.21.15.28:0/2662993091 >> v2:172.21.15.191:3300/0 conn(0x7fc0980c42f0 msgr2=0x7fc0980c3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:49:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.592+0000 7fc097e00640 1 --2- 172.21.15.28:0/2662993091 >> v2:172.21.15.191:3300/0 conn(0x7fc0980c42f0 0x7fc0980c3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.592+0000 7fc097e00640 1 -- 172.21.15.28:0/2662993091 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc098156070 con 0x7fc09814f790 2024-09-13T16:49:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.593+0000 7fc097e00640 1 --2- 172.21.15.28:0/2662993091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc09814f790 0x7fc0980cabd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc0980c7f70 tx=0x7fc08802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc725fcf201a0109 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.593+0000 7fc096a00640 1 -- 172.21.15.28:0/2662993091 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088032440 con 0x7fc09814f790 2024-09-13T16:49:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.593+0000 7fc096a00640 1 -- 172.21.15.28:0/2662993091 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fc0880325a0 con 0x7fc09814f790 2024-09-13T16:49:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.593+0000 7fc096a00640 1 -- 172.21.15.28:0/2662993091 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088032950 con 0x7fc09814f790 2024-09-13T16:49:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.594+0000 7fc09da00640 1 -- 172.21.15.28:0/2662993091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc09814f790 msgr2=0x7fc0980cabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.594+0000 7fc09da00640 1 --2- 172.21.15.28:0/2662993091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc09814f790 0x7fc0980cabd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc0980c7f70 tx=0x7fc08802a270 comp rx=0 tx=0).stop 2024-09-13T16:49:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.594+0000 7fc09da00640 1 -- 172.21.15.28:0/2662993091 shutdown_connections 2024-09-13T16:49:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.594+0000 7fc09da00640 1 --2- 172.21.15.28:0/2662993091 >> v2:172.21.15.191:3300/0 conn(0x7fc0980c42f0 0x7fc0980c3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.594+0000 7fc09da00640 1 --2- 172.21.15.28:0/2662993091 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc09814fb70 0x7fc0980c3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.594+0000 7fc09da00640 1 --2- 172.21.15.28:0/2662993091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc09814f790 0x7fc0980cabd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.594+0000 7fc09da00640 1 -- 172.21.15.28:0/2662993091 >> 172.21.15.28:0/2662993091 conn(0x7fc0980ca0e0 msgr2=0x7fc0980ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.594+0000 7fc09da00640 1 -- 172.21.15.28:0/2662993091 shutdown_connections 2024-09-13T16:49:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.594+0000 7fc09da00640 1 -- 172.21.15.28:0/2662993091 wait complete. 2024-09-13T16:49:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.596+0000 7fc09da00640 1 Processor -- start 2024-09-13T16:49:12.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.596+0000 7fc09da00640 1 -- start start 2024-09-13T16:49:12.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.597+0000 7fc09da00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0980c42f0 0x7fc0980c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.597+0000 7fc09da00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc09814f790 0x7fc098147350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.597+0000 7fc09da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc098147890 0x7fc098143d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.597+0000 7fc09da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09814c9c0 con 0x7fc098147890 2024-09-13T16:49:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.597+0000 7fc09da00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc098155150 con 0x7fc09814f790 2024-09-13T16:49:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.597+0000 7fc09da00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09814a9b0 con 0x7fc0980c42f0 2024-09-13T16:49:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.597+0000 7fc09d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc098147890 0x7fc098143d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.597+0000 7fc09d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc098147890 0x7fc098143d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35688/0 (socket says 172.21.15.28:35688) 2024-09-13T16:49:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.597+0000 7fc09d000640 1 -- 172.21.15.28:0/3880050442 learned_addr learned my addr 172.21.15.28:0/3880050442 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:49:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.597+0000 7fc097e00640 1 --2- 172.21.15.28:0/3880050442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0980c42f0 0x7fc0980c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.597+0000 7fc097400640 1 --2- 172.21.15.28:0/3880050442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc09814f790 0x7fc098147350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.597+0000 7fc09d000640 1 -- 172.21.15.28:0/3880050442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0980c42f0 msgr2=0x7fc0980c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.597+0000 7fc09d000640 1 --2- 172.21.15.28:0/3880050442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0980c42f0 0x7fc0980c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.597+0000 7fc09d000640 1 -- 172.21.15.28:0/3880050442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc09814f790 msgr2=0x7fc098147350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.597+0000 7fc09d000640 1 --2- 172.21.15.28:0/3880050442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc09814f790 0x7fc098147350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.597+0000 7fc09d000640 1 -- 172.21.15.28:0/3880050442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0880094d0 con 0x7fc098147890 2024-09-13T16:49:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.598+0000 7fc09d000640 1 --2- 172.21.15.28:0/3880050442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc098147890 0x7fc098143d70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc09000eed0 tx=0x7fc09000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.598+0000 7fc08fe00640 1 -- 172.21.15.28:0/3880050442 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc090010040 con 0x7fc098147890 2024-09-13T16:49:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.598+0000 7fc08fe00640 1 -- 172.21.15.28:0/3880050442 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fc090014400 con 0x7fc098147890 2024-09-13T16:49:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.598+0000 7fc08fe00640 1 -- 172.21.15.28:0/3880050442 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0900147d0 con 0x7fc098147890 2024-09-13T16:49:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.598+0000 7fc09da00640 1 -- 172.21.15.28:0/3880050442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0981553a0 con 0x7fc098147890 2024-09-13T16:49:12.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.598+0000 7fc09da00640 1 -- 172.21.15.28:0/3880050442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09814d620 con 0x7fc098147890 2024-09-13T16:49:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.599+0000 7fc09da00640 1 -- 172.21.15.28:0/3880050442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0980cabd0 con 0x7fc098147890 2024-09-13T16:49:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.600+0000 7fc08fe00640 1 -- 172.21.15.28:0/3880050442 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc090014930 con 0x7fc098147890 2024-09-13T16:49:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.601+0000 7fc08fe00640 1 --2- 172.21.15.28:0/3880050442 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc07806ac30 0x7fc07806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.601+0000 7fc08fe00640 1 -- 172.21.15.28:0/3880050442 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc090089f10 con 0x7fc098147890 2024-09-13T16:49:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.601+0000 7fc097e00640 1 --2- 172.21.15.28:0/3880050442 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc07806ac30 0x7fc07806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.602+0000 7fc097e00640 1 --2- 172.21.15.28:0/3880050442 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc07806ac30 0x7fc07806d0f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc088034000 tx=0x7fc08802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.605+0000 7fc08fe00640 1 -- 172.21.15.28:0/3880050442 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc090054f60 con 0x7fc098147890 2024-09-13T16:49:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.786+0000 7fc09da00640 1 -- 172.21.15.28:0/3880050442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc098070b10 con 0x7fc098147890 2024-09-13T16:49:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.787+0000 7fc08fe00640 1 -- 172.21.15.28:0/3880050442 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16673 (secure 0 0 0) 0x7fc090018070 con 0x7fc098147890 2024-09-13T16:49:12.789 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:49:12.790 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":22,"fsid":"8e702092-2b7f-4633-9ce6-17040007e53d","created":"2024-09-13T16:43:39.459348+0000","modified":"2024-09-13T16:44:58.950267+0000","last_up_change":"2024-09-13T16:43:49.476313+0000","last_in_change":"2024-09-13T16:43:41.595256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:43:49.643606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:43:52.828510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:43:53.945874+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274e98e1-663e-4814-90ad-c7b3026b276d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6801","nonce":2964323435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6803","nonce":2964323435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6807","nonce":2964323435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2964323435},{"type":"v1","addr":"172.21.15.28:6805","nonce":2964323435}]},"public_addr":"172.21.15.28:6801/2964323435","cluster_addr":"172.21.15.28:6803/2964323435","heartbeat_back_addr":"172.21.15.28:6807/2964323435","heartbeat_front_addr":"172.21.15.28:6805/2964323435","state":["exists","up"]},{"osd":1,"uuid":"67a0a7cd-ee9b-471c-a34f-d4fe97186aa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6809","nonce":23768570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6811","nonce":23768570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6815","nonce":23768570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":23768570},{"type":"v1","addr":"172.21.15.28:6813","nonce":23768570}]},"public_addr":"172.21.15.28:6809/23768570","cluster_addr":"172.21.15.28:6811/23768570","heartbeat_back_addr":"172.21.15.28:6815/23768570","heartbeat_front_addr":"172.21.15.28:6813/23768570","state":["exists","up"]},{"osd":2,"uuid":"d3cd6efd-ff5f-4826-822c-79ebd46af2e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6817","nonce":886887303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6819","nonce":886887303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6823","nonce":886887303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":886887303},{"type":"v1","addr":"172.21.15.28:6821","nonce":886887303}]},"public_addr":"172.21.15.28:6817/886887303","cluster_addr":"172.21.15.28:6819/886887303","heartbeat_back_addr":"172.21.15.28:6823/886887303","heartbeat_front_addr":"172.21.15.28:6821/886887303","state":["exists","up"]},{"osd":3,"uuid":"5f457889-9f7e-4799-8dc3-d58725734ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6825","nonce":2616036674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6827","nonce":2616036674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6831","nonce":2616036674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2616036674},{"type":"v1","addr":"172.21.15.28:6829","nonce":2616036674}]},"public_addr":"172.21.15.28:6825/2616036674","cluster_addr":"172.21.15.28:6827/2616036674","heartbeat_back_addr":"172.21.15.28:6831/2616036674","heartbeat_front_addr":"172.21.15.28:6829/2616036674","state":["exists","up"]},{"osd":4,"uuid":"5cc0d9dd-9486-4893-b94a-7dfdf32edbee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6801","nonce":2817102450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6803","nonce":2817102450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6807","nonce":2817102450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2817102450},{"type":"v1","addr":"172.21.15.191:6805","nonce":2817102450}]},"public_addr":"172.21.15.191:6801/2817102450","cluster_addr":"172.21.15.191:6803/2817102450","heartbeat_back_addr":"172.21.15.191:6807/2817102450","heartbeat_front_addr":"172.21.15.191:6805/2817102450","state":["exists","up"]},{"osd":5,"uuid":"fabc87ad-109e-4b4c-8f9a-8d6ccfd89c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6817","nonce":2503027971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6819","nonce":2503027971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6823","nonce":2503027971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2503027971},{"type":"v1","addr":"172.21.15.191:6821","nonce":2503027971}]},"public_addr":"172.21.15.191:6817/2503027971","cluster_addr":"172.21.15.191:6819/2503027971","heartbeat_back_addr":"172.21.15.191:6823/2503027971","heartbeat_front_addr":"172.21.15.191:6821/2503027971","state":["exists","up"]},{"osd":6,"uuid":"53a405e2-64d2-4baa-9a8c-16b03368865c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6809","nonce":3006595088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6811","nonce":3006595088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6815","nonce":3006595088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3006595088},{"type":"v1","addr":"172.21.15.191:6813","nonce":3006595088}]},"public_addr":"172.21.15.191:6809/3006595088","cluster_addr":"172.21.15.191:6811/3006595088","heartbeat_back_addr":"172.21.15.191:6815/3006595088","heartbeat_front_addr":"172.21.15.191:6813/3006595088","state":["exists","up"]},{"osd":7,"uuid":"001391e2-0bf7-4397-9a21-f44efea6b269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6825","nonce":1477327720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6827","nonce":1477327720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6831","nonce":1477327720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1477327720},{"type":"v1","addr":"172.21.15.191:6829","nonce":1477327720}]},"public_addr":"172.21.15.191:6825/1477327720","cluster_addr":"172.21.15.191:6827/1477327720","heartbeat_back_addr":"172.21.15.191:6831/1477327720","heartbeat_front_addr":"172.21.15.191:6829/1477327720","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.917250+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.954351+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.017329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:47.046221+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.707467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.726947+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.782321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:43:46.764850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.1c","mappings":[{"from":5,"to":1}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:49:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.790+0000 7fc09da00640 1 -- 172.21.15.28:0/3880050442 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc07806ac30 msgr2=0x7fc07806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.790+0000 7fc09da00640 1 --2- 172.21.15.28:0/3880050442 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc07806ac30 0x7fc07806d0f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc088034000 tx=0x7fc08802e000 comp rx=0 tx=0).stop 2024-09-13T16:49:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.790+0000 7fc09da00640 1 -- 172.21.15.28:0/3880050442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc098147890 msgr2=0x7fc098143d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.790+0000 7fc09da00640 1 --2- 172.21.15.28:0/3880050442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc098147890 0x7fc098143d70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc09000eed0 tx=0x7fc09000b480 comp rx=0 tx=0).stop 2024-09-13T16:49:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.790+0000 7fc09da00640 1 -- 172.21.15.28:0/3880050442 shutdown_connections 2024-09-13T16:49:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.790+0000 7fc09da00640 1 --2- 172.21.15.28:0/3880050442 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fc07806ac30 0x7fc07806d0f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.790+0000 7fc09da00640 1 --2- 172.21.15.28:0/3880050442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc098147890 0x7fc098143d70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.790+0000 7fc09da00640 1 --2- 172.21.15.28:0/3880050442 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc09814f790 0x7fc098147350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.790+0000 7fc09da00640 1 --2- 172.21.15.28:0/3880050442 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0980c42f0 0x7fc0980c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.790+0000 7fc09da00640 1 -- 172.21.15.28:0/3880050442 >> 172.21.15.28:0/3880050442 conn(0x7fc0980ca0e0 msgr2=0x7fc0980c30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.790+0000 7fc09da00640 1 -- 172.21.15.28:0/3880050442 shutdown_connections 2024-09-13T16:49:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.790+0000 7fc09da00640 1 -- 172.21.15.28:0/3880050442 wait complete. 2024-09-13T16:49:12.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T16:49:12.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.963+0000 7fa536000640 1 Processor -- start 2024-09-13T16:49:12.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.963+0000 7fa536000640 1 -- start start 2024-09-13T16:49:12.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.964+0000 7fa536000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa5300c5bd0 0x7fa530156150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.964+0000 7fa536000640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa5300c6ef0 0x7fa5300c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.964+0000 7fa536000640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53014ca00 con 0x7fa5300c3670 2024-09-13T16:49:12.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.964+0000 7fa536000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5300c2dc0 con 0x7fa5300c5bd0 2024-09-13T16:49:12.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.964+0000 7fa536000640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa530155780 con 0x7fa5300c6ef0 2024-09-13T16:49:12.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.964+0000 7fa52f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa5300c5bd0 0x7fa530156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.964+0000 7fa52fe00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa5300c6ef0 0x7fa5300c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.965+0000 7fa52f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa5300c5bd0 0x7fa530156150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35694/0 (socket says 172.21.15.28:35694) 2024-09-13T16:49:12.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.965+0000 7fa52f400640 1 -- 172.21.15.28:0/2868666390 learned_addr learned my addr 172.21.15.28:0/2868666390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:49:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.965+0000 7fa52f400640 1 -- 172.21.15.28:0/2868666390 >> v2:172.21.15.191:3300/0 conn(0x7fa5300c6ef0 msgr2=0x7fa5300c73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:49:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.965+0000 7fa52f400640 1 --2- 172.21.15.28:0/2868666390 >> v2:172.21.15.191:3300/0 conn(0x7fa5300c6ef0 0x7fa5300c73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.965+0000 7fa52f400640 1 -- 172.21.15.28:0/2868666390 >> v1:172.21.15.191:6789/0 conn(0x7fa5300c3670 legacy=0x7fa5300c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.965+0000 7fa52f400640 1 -- 172.21.15.28:0/2868666390 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa530155990 con 0x7fa5300c5bd0 2024-09-13T16:49:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.966+0000 7fa52f400640 1 --2- 172.21.15.28:0/2868666390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5300c5bd0 0x7fa530156150 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa530072590 tx=0x7fa52002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a822de1b71bf001b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.966+0000 7fa52e000640 1 -- 172.21.15.28:0/2868666390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa520032640 con 0x7fa5300c5bd0 2024-09-13T16:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.966+0000 7fa52e000640 1 -- 172.21.15.28:0/2868666390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fa5200327a0 con 0x7fa5300c5bd0 2024-09-13T16:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.966+0000 7fa52e000640 1 -- 172.21.15.28:0/2868666390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa520032b50 con 0x7fa5300c5bd0 2024-09-13T16:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.967+0000 7fa536000640 1 -- 172.21.15.28:0/2868666390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5300c5bd0 msgr2=0x7fa530156150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.967+0000 7fa536000640 1 --2- 172.21.15.28:0/2868666390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5300c5bd0 0x7fa530156150 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa530072590 tx=0x7fa52002a300 comp rx=0 tx=0).stop 2024-09-13T16:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.967+0000 7fa536000640 1 -- 172.21.15.28:0/2868666390 shutdown_connections 2024-09-13T16:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.967+0000 7fa536000640 1 --2- 172.21.15.28:0/2868666390 >> v2:172.21.15.191:3300/0 conn(0x7fa5300c6ef0 0x7fa5300c73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.967+0000 7fa536000640 1 --2- 172.21.15.28:0/2868666390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5300c5bd0 0x7fa530156150 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.967+0000 7fa536000640 1 -- 172.21.15.28:0/2868666390 >> 172.21.15.28:0/2868666390 conn(0x7fa530066040 msgr2=0x7fa530069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.967+0000 7fa536000640 1 -- 172.21.15.28:0/2868666390 shutdown_connections 2024-09-13T16:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.967+0000 7fa536000640 1 -- 172.21.15.28:0/2868666390 wait complete. 2024-09-13T16:49:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.968+0000 7fa536000640 1 Processor -- start 2024-09-13T16:49:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.968+0000 7fa536000640 1 -- start start 2024-09-13T16:49:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.969+0000 7fa536000640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5300c3670 0x7fa5301eed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.969+0000 7fa536000640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5300c5bd0 0x7fa5301e0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.969+0000 7fa536000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5300c6ef0 0x7fa53014bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.969+0000 7fa536000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5301ee380 con 0x7fa5300c6ef0 2024-09-13T16:49:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.969+0000 7fa536000640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5301ee900 con 0x7fa5300c3670 2024-09-13T16:49:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.969+0000 7fa536000640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5301ddf90 con 0x7fa5300c5bd0 2024-09-13T16:49:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.970+0000 7fa52fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5300c6ef0 0x7fa53014bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.970+0000 7fa52fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5300c6ef0 0x7fa53014bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35702/0 (socket says 172.21.15.28:35702) 2024-09-13T16:49:12.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.970+0000 7fa52fe00640 1 -- 172.21.15.28:0/1329715183 learned_addr learned my addr 172.21.15.28:0/1329715183 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:49:12.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.970+0000 7fa52f400640 1 --2- 172.21.15.28:0/1329715183 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5300c3670 0x7fa5301eed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.970+0000 7fa52ea00640 1 --2- 172.21.15.28:0/1329715183 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5300c5bd0 0x7fa5301e0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.970+0000 7fa52fe00640 1 -- 172.21.15.28:0/1329715183 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5300c5bd0 msgr2=0x7fa5301e0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.970+0000 7fa52fe00640 1 --2- 172.21.15.28:0/1329715183 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5300c5bd0 0x7fa5301e0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.970+0000 7fa52fe00640 1 -- 172.21.15.28:0/1329715183 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5300c3670 msgr2=0x7fa5301eed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.970+0000 7fa52fe00640 1 --2- 172.21.15.28:0/1329715183 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5300c3670 0x7fa5301eed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.970+0000 7fa52fe00640 1 -- 172.21.15.28:0/1329715183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5200094d0 con 0x7fa5300c6ef0 2024-09-13T16:49:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.970+0000 7fa52fe00640 1 --2- 172.21.15.28:0/1329715183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5300c6ef0 0x7fa53014bdb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa52800efc0 tx=0x7fa52800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.971+0000 7fa527400640 1 -- 172.21.15.28:0/1329715183 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa528009ba0 con 0x7fa5300c6ef0 2024-09-13T16:49:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.971+0000 7fa527400640 1 -- 172.21.15.28:0/1329715183 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fa52800f040 con 0x7fa5300c6ef0 2024-09-13T16:49:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.971+0000 7fa527400640 1 -- 172.21.15.28:0/1329715183 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5280146e0 con 0x7fa5300c6ef0 2024-09-13T16:49:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.971+0000 7fa536000640 1 -- 172.21.15.28:0/1329715183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5301ef2d0 con 0x7fa5300c6ef0 2024-09-13T16:49:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.971+0000 7fa536000640 1 -- 172.21.15.28:0/1329715183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5301e1eb0 con 0x7fa5300c6ef0 2024-09-13T16:49:12.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.972+0000 7fa536000640 1 -- 172.21.15.28:0/1329715183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5301e20d0 con 0x7fa5300c6ef0 2024-09-13T16:49:12.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.973+0000 7fa527400640 1 -- 172.21.15.28:0/1329715183 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa528014840 con 0x7fa5300c6ef0 2024-09-13T16:49:12.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.973+0000 7fa527400640 1 --2- 172.21.15.28:0/1329715183 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa51006ac00 0x7fa51006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:12.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.974+0000 7fa527400640 1 -- 172.21.15.28:0/1329715183 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa528089ed0 con 0x7fa5300c6ef0 2024-09-13T16:49:12.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.974+0000 7fa52f400640 1 --2- 172.21.15.28:0/1329715183 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa51006ac00 0x7fa51006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:12.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.974+0000 7fa52f400640 1 --2- 172.21.15.28:0/1329715183 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa51006ac00 0x7fa51006d0c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa520034000 tx=0x7fa52002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:12.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:12.978+0000 7fa527400640 1 -- 172.21.15.28:0/1329715183 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa528054fb0 con 0x7fa5300c6ef0 2024-09-13T16:49:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.191+0000 7fa536000640 1 -- 172.21.15.28:0/1329715183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa5301eea70 con 0x7fa5300c6ef0 2024-09-13T16:49:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.203+0000 7fa527400640 1 -- 172.21.15.28:0/1329715183 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa528018070 con 0x7fa5300c6ef0 2024-09-13T16:49:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T16:49:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.205+0000 7fa536000640 1 -- 172.21.15.28:0/1329715183 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa51006ac00 msgr2=0x7fa51006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.205+0000 7fa536000640 1 --2- 172.21.15.28:0/1329715183 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa51006ac00 0x7fa51006d0c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa520034000 tx=0x7fa52002e000 comp rx=0 tx=0).stop 2024-09-13T16:49:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.205+0000 7fa536000640 1 -- 172.21.15.28:0/1329715183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5300c6ef0 msgr2=0x7fa53014bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.205+0000 7fa536000640 1 --2- 172.21.15.28:0/1329715183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5300c6ef0 0x7fa53014bdb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa52800efc0 tx=0x7fa52800c490 comp rx=0 tx=0).stop 2024-09-13T16:49:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.205+0000 7fa536000640 1 -- 172.21.15.28:0/1329715183 shutdown_connections 2024-09-13T16:49:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.205+0000 7fa536000640 1 --2- 172.21.15.28:0/1329715183 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fa51006ac00 0x7fa51006d0c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.205+0000 7fa536000640 1 --2- 172.21.15.28:0/1329715183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5300c6ef0 0x7fa53014bdb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.205+0000 7fa536000640 1 --2- 172.21.15.28:0/1329715183 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5300c5bd0 0x7fa5301e0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.205+0000 7fa536000640 1 --2- 172.21.15.28:0/1329715183 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5300c3670 0x7fa5301eed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.205+0000 7fa536000640 1 -- 172.21.15.28:0/1329715183 >> 172.21.15.28:0/1329715183 conn(0x7fa530066040 msgr2=0x7fa5300c8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.205+0000 7fa536000640 1 -- 172.21.15.28:0/1329715183 shutdown_connections 2024-09-13T16:49:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.205+0000 7fa536000640 1 -- 172.21.15.28:0/1329715183 wait complete. 2024-09-13T16:49:13.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T16:49:13.246 INFO:tasks.ceph.mds.f.smithi191.stdout:starting mds.f at 2024-09-13T16:49:13.246 INFO:tasks.ceph.mds.b.smithi191.stdout:starting mds.b at 2024-09-13T16:49:13.250 INFO:tasks.ceph.mds.d.smithi191.stdout:starting mds.d at 2024-09-13T16:49:13.270 INFO:tasks.ceph.mds.e.smithi028.stdout:starting mds.e at 2024-09-13T16:49:13.280 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T16:49:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.391+0000 7f619be00640 1 Processor -- start 2024-09-13T16:49:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.392+0000 7f619be00640 1 -- start start 2024-09-13T16:49:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.393+0000 7f619be00640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f619c0c7af0 0x7f619c15c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.393+0000 7f619be00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f619c0c5b80 0x7f619c152590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.393+0000 7f619be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619c15cdc0 con 0x7f619c0c7710 2024-09-13T16:49:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.393+0000 7f619be00640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619c070b60 con 0x7f619c0c7af0 2024-09-13T16:49:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.393+0000 7f619be00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619c0c6150 con 0x7f619c0c5b80 2024-09-13T16:49:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.393+0000 7f619aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f619c0c7710 0x7f619c070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55266/0 (socket says 172.21.15.28:55266) 2024-09-13T16:49:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.393+0000 7f619aa00640 1 -- 172.21.15.28:0/3555298842 learned_addr learned my addr 172.21.15.28:0/3555298842 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:49:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.393+0000 7f619b400640 1 --2- 172.21.15.28:0/3555298842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f619c0c5b80 0x7f619c152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.393+0000 7f6193e00640 1 --2- 172.21.15.28:0/3555298842 >> v2:172.21.15.191:3300/0 conn(0x7f619c0c7af0 0x7f619c15c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.394+0000 7f6193e00640 1 -- 172.21.15.28:0/3555298842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f619c0c5b80 msgr2=0x7f619c152590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:49:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.394+0000 7f6193e00640 1 --2- 172.21.15.28:0/3555298842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f619c0c5b80 0x7f619c152590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.394+0000 7f6193e00640 1 -- 172.21.15.28:0/3555298842 >> v1:172.21.15.28:6789/0 conn(0x7f619c0c7710 legacy=0x7f619c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.394+0000 7f619a000640 1 -- 172.21.15.28:0/3555298842 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050621987 0 0) 0x7f618c001110 con 0x7f619c0c7710 2024-09-13T16:49:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.394+0000 7f6193e00640 1 -- 172.21.15.28:0/3555298842 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619c0c6430 con 0x7f619c0c7af0 2024-09-13T16:49:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.394+0000 7f619b400640 1 --2- 172.21.15.28:0/3555298842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f619c0c5b80 0x7f619c152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:49:13.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.395+0000 7f6193e00640 1 --2- 172.21.15.28:0/3555298842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f619c0c7af0 0x7f619c15c880 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f61840097a0 tx=0x7f618402a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8bfb722a224f5e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:13.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.395+0000 7f619a000640 1 -- 172.21.15.28:0/3555298842 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6184032440 con 0x7f619c0c7af0 2024-09-13T16:49:13.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.395+0000 7f619a000640 1 -- 172.21.15.28:0/3555298842 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f61840325a0 con 0x7f619c0c7af0 2024-09-13T16:49:13.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.395+0000 7f619a000640 1 -- 172.21.15.28:0/3555298842 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6184032950 con 0x7f619c0c7af0 2024-09-13T16:49:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.395+0000 7f619be00640 1 -- 172.21.15.28:0/3555298842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f619c0c7af0 msgr2=0x7f619c15c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.395+0000 7f619be00640 1 --2- 172.21.15.28:0/3555298842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f619c0c7af0 0x7f619c15c880 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f61840097a0 tx=0x7f618402a210 comp rx=0 tx=0).stop 2024-09-13T16:49:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.396+0000 7f619be00640 1 -- 172.21.15.28:0/3555298842 shutdown_connections 2024-09-13T16:49:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.396+0000 7f619be00640 1 --2- 172.21.15.28:0/3555298842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f619c0c5b80 0x7f619c152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.396+0000 7f619be00640 1 --2- 172.21.15.28:0/3555298842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f619c0c7af0 0x7f619c15c880 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.396+0000 7f619be00640 1 -- 172.21.15.28:0/3555298842 >> 172.21.15.28:0/3555298842 conn(0x7f619c06e220 msgr2=0x7f619c06e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.396+0000 7f619be00640 1 -- 172.21.15.28:0/3555298842 shutdown_connections 2024-09-13T16:49:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.396+0000 7f619be00640 1 -- 172.21.15.28:0/3555298842 wait complete. 2024-09-13T16:49:13.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.397+0000 7f619be00640 1 Processor -- start 2024-09-13T16:49:13.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.397+0000 7f619be00640 1 -- start start 2024-09-13T16:49:13.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.398+0000 7f619be00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f619c0c5b80 0x7f619c0c4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:13.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.398+0000 7f619be00640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f619c0c7710 0x7f619c0cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:13.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.398+0000 7f619be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c0c7af0 0x7f619c15ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:13.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.398+0000 7f619be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c0c2ee0 con 0x7f619c0c7af0 2024-09-13T16:49:13.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.398+0000 7f619be00640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c0c4b60 con 0x7f619c0c7710 2024-09-13T16:49:13.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.398+0000 7f619be00640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619c0cb610 con 0x7f619c0c5b80 2024-09-13T16:49:13.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.398+0000 7f619b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c0c7af0 0x7f619c15ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:13.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.398+0000 7f619b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c0c7af0 0x7f619c15ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35740/0 (socket says 172.21.15.28:35740) 2024-09-13T16:49:13.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.398+0000 7f619b400640 1 -- 172.21.15.28:0/1545761470 learned_addr learned my addr 172.21.15.28:0/1545761470 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:49:13.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.398+0000 7f6193e00640 1 --2- 172.21.15.28:0/1545761470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f619c0c7710 0x7f619c0cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:13.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.398+0000 7f619aa00640 1 --2- 172.21.15.28:0/1545761470 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f619c0c5b80 0x7f619c0c4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:13.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.399+0000 7f619b400640 1 -- 172.21.15.28:0/1545761470 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f619c0c5b80 msgr2=0x7f619c0c4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:13.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.399+0000 7f619b400640 1 --2- 172.21.15.28:0/1545761470 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f619c0c5b80 0x7f619c0c4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:13.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.399+0000 7f619b400640 1 -- 172.21.15.28:0/1545761470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f619c0c7710 msgr2=0x7f619c0cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:13.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.399+0000 7f619b400640 1 --2- 172.21.15.28:0/1545761470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f619c0c7710 0x7f619c0cb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:13.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.399+0000 7f619b400640 1 -- 172.21.15.28:0/1545761470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6194009a70 con 0x7f619c0c7af0 2024-09-13T16:49:13.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.399+0000 7f6193e00640 1 --2- 172.21.15.28:0/1545761470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f619c0c7710 0x7f619c0cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:49:13.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.399+0000 7f619b400640 1 --2- 172.21.15.28:0/1545761470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c0c7af0 0x7f619c15ad20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6194009f00 tx=0x7f619400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:13.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.399+0000 7f6192a00640 1 -- 172.21.15.28:0/1545761470 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194009d70 con 0x7f619c0c7af0 2024-09-13T16:49:13.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.399+0000 7f6192a00640 1 -- 172.21.15.28:0/1545761470 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f619400f040 con 0x7f619c0c7af0 2024-09-13T16:49:13.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.399+0000 7f619be00640 1 -- 172.21.15.28:0/1545761470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6184009400 con 0x7f619c0c7af0 2024-09-13T16:49:13.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.399+0000 7f6192a00640 1 -- 172.21.15.28:0/1545761470 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194003bd0 con 0x7f619c0c7af0 2024-09-13T16:49:13.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.400+0000 7f619be00640 1 -- 172.21.15.28:0/1545761470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619c0c33f0 con 0x7f619c0c7af0 2024-09-13T16:49:13.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.401+0000 7f619be00640 1 -- 172.21.15.28:0/1545761470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619c1596f0 con 0x7f619c0c7af0 2024-09-13T16:49:13.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.402+0000 7f6192a00640 1 -- 172.21.15.28:0/1545761470 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6194014480 con 0x7f619c0c7af0 2024-09-13T16:49:13.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.403+0000 7f6192a00640 1 --2- 172.21.15.28:0/1545761470 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f617806ac00 0x7f617806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:13.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.403+0000 7f6192a00640 1 -- 172.21.15.28:0/1545761470 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f619408a1d0 con 0x7f619c0c7af0 2024-09-13T16:49:13.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.403+0000 7f619aa00640 1 --2- 172.21.15.28:0/1545761470 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f617806ac00 0x7f617806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:13.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.404+0000 7f619aa00640 1 --2- 172.21.15.28:0/1545761470 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f617806ac00 0x7f617806d0c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f618c000f60 tx=0x7f618c007b90 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.408+0000 7f6192a00640 1 -- 172.21.15.28:0/1545761470 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6194055110 con 0x7f619c0c7af0 2024-09-13T16:49:13.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:13.621+0000 7f619be00640 1 -- 172.21.15.28:0/1545761470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f619c0673d0 con 0x7f619c0c7af0 2024-09-13T16:49:14.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.205+0000 7f6192a00640 1 -- 172.21.15.28:0/1545761470 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6194058730 con 0x7f619c0c7af0 2024-09-13T16:49:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.208+0000 7f619be00640 1 -- 172.21.15.28:0/1545761470 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f617806ac00 msgr2=0x7f617806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.208+0000 7f619be00640 1 --2- 172.21.15.28:0/1545761470 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f617806ac00 0x7f617806d0c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f618c000f60 tx=0x7f618c007b90 comp rx=0 tx=0).stop 2024-09-13T16:49:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.208+0000 7f619be00640 1 -- 172.21.15.28:0/1545761470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c0c7af0 msgr2=0x7f619c15ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.208+0000 7f619be00640 1 --2- 172.21.15.28:0/1545761470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c0c7af0 0x7f619c15ad20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6194009f00 tx=0x7f619400c490 comp rx=0 tx=0).stop 2024-09-13T16:49:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.208+0000 7f619be00640 1 -- 172.21.15.28:0/1545761470 shutdown_connections 2024-09-13T16:49:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.208+0000 7f619be00640 1 --2- 172.21.15.28:0/1545761470 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7f617806ac00 0x7f617806d0c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.208+0000 7f619be00640 1 --2- 172.21.15.28:0/1545761470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619c0c7af0 0x7f619c15ad20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.208+0000 7f619be00640 1 --2- 172.21.15.28:0/1545761470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f619c0c7710 0x7f619c0cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.208+0000 7f619be00640 1 --2- 172.21.15.28:0/1545761470 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f619c0c5b80 0x7f619c0c4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.208+0000 7f619be00640 1 -- 172.21.15.28:0/1545761470 >> 172.21.15.28:0/1545761470 conn(0x7f619c06e220 msgr2=0x7f619c157050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.208+0000 7f619be00640 1 -- 172.21.15.28:0/1545761470 shutdown_connections 2024-09-13T16:49:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.208+0000 7f619be00640 1 -- 172.21.15.28:0/1545761470 wait complete. 2024-09-13T16:49:14.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T16:49:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.432+0000 7fe3c2800640 1 Processor -- start 2024-09-13T16:49:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.433+0000 7fe3c2800640 1 -- start start 2024-09-13T16:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.434+0000 7fe3c2800640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe3bc0c6240 0x7fe3bc162010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.434+0000 7fe3c2800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3bc06e100 con 0x7fe3bc0c1b40 2024-09-13T16:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.434+0000 7fe3c2800640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3bc150d90 con 0x7fe3bc1509b0 2024-09-13T16:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.434+0000 7fe3c2800640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3bc0c43b0 con 0x7fe3bc0c6240 2024-09-13T16:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.434+0000 7fe3c0a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe3bc0c1b40 0x7fe3bc156990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55272/0 (socket says 172.21.15.28:55272) 2024-09-13T16:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.434+0000 7fe3c0a00640 1 -- 172.21.15.28:0/290980841 learned_addr learned my addr 172.21.15.28:0/290980841 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T16:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.434+0000 7fe3c1e00640 1 --2- 172.21.15.28:0/290980841 >> v2:172.21.15.191:3300/0 conn(0x7fe3bc0c6240 0x7fe3bc162010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.435+0000 7fe3bbe00640 1 -- 172.21.15.28:0/290980841 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739121805 0 0) 0x7fe3bc06e100 con 0x7fe3bc0c1b40 2024-09-13T16:49:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.435+0000 7fe3bbe00640 1 -- 172.21.15.28:0/290980841 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a4003520 con 0x7fe3bc0c1b40 2024-09-13T16:49:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.435+0000 7fe3c1e00640 1 -- 172.21.15.28:0/290980841 >> v1:172.21.15.191:6789/0 conn(0x7fe3bc1509b0 legacy=0x7fe3bc06cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.435+0000 7fe3c1e00640 1 -- 172.21.15.28:0/290980841 >> v1:172.21.15.28:6789/0 conn(0x7fe3bc0c1b40 legacy=0x7fe3bc156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.435+0000 7fe3bbe00640 1 -- 172.21.15.28:0/290980841 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759318840 0 0) 0x7fe3bc150d90 con 0x7fe3bc1509b0 2024-09-13T16:49:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.435+0000 7fe3c1e00640 1 -- 172.21.15.28:0/290980841 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc06d7d0 con 0x7fe3bc0c6240 2024-09-13T16:49:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.436+0000 7fe3c1e00640 1 --2- 172.21.15.28:0/290980841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3bc0c6240 0x7fe3bc162010 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009d90 tx=0x7fe3b402a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5f0e54e3cfbe9f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.436+0000 7fe3bbe00640 1 -- 172.21.15.28:0/290980841 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4007740 con 0x7fe3bc0c6240 2024-09-13T16:49:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.436+0000 7fe3bbe00640 1 -- 172.21.15.28:0/290980841 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe3b402d050 con 0x7fe3bc0c6240 2024-09-13T16:49:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.436+0000 7fe3bbe00640 1 -- 172.21.15.28:0/290980841 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4007aa0 con 0x7fe3bc0c6240 2024-09-13T16:49:14.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.437+0000 7fe3c2800640 1 -- 172.21.15.28:0/290980841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3bc0c6240 msgr2=0x7fe3bc162010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:14.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.437+0000 7fe3c2800640 1 --2- 172.21.15.28:0/290980841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3bc0c6240 0x7fe3bc162010 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009d90 tx=0x7fe3b402a6f0 comp rx=0 tx=0).stop 2024-09-13T16:49:14.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.437+0000 7fe3c2800640 1 -- 172.21.15.28:0/290980841 shutdown_connections 2024-09-13T16:49:14.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.437+0000 7fe3c2800640 1 --2- 172.21.15.28:0/290980841 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3bc0c6240 0x7fe3bc162010 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:14.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.437+0000 7fe3c2800640 1 -- 172.21.15.28:0/290980841 >> 172.21.15.28:0/290980841 conn(0x7fe3bc0ca0e0 msgr2=0x7fe3bc0ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:14.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.437+0000 7fe3c2800640 1 -- 172.21.15.28:0/290980841 shutdown_connections 2024-09-13T16:49:14.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.437+0000 7fe3c2800640 1 -- 172.21.15.28:0/290980841 wait complete. 2024-09-13T16:49:14.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.439+0000 7fe3c2800640 1 Processor -- start 2024-09-13T16:49:14.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.439+0000 7fe3c2800640 1 -- start start 2024-09-13T16:49:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.440+0000 7fe3c2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0c1b40 0x7fe3bc143110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.440+0000 7fe3c2800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3bc0c6240 0x7fe3bc13daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.440+0000 7fe3c2800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3bc1509b0 0x7fe3bc13e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.440+0000 7fe3c2800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc070d10 con 0x7fe3bc0c1b40 2024-09-13T16:49:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.440+0000 7fe3c2800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc071ba0 con 0x7fe3bc0c6240 2024-09-13T16:49:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.440+0000 7fe3c2800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc070a50 con 0x7fe3bc1509b0 2024-09-13T16:49:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.440+0000 7fe3c1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0c1b40 0x7fe3bc143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.440+0000 7fe3c1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0c1b40 0x7fe3bc143110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35750/0 (socket says 172.21.15.28:35750) 2024-09-13T16:49:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.440+0000 7fe3c1400640 1 -- 172.21.15.28:0/1728140640 learned_addr learned my addr 172.21.15.28:0/1728140640 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:49:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.440+0000 7fe3c0a00640 1 --2- 172.21.15.28:0/1728140640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3bc0c6240 0x7fe3bc13daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.440+0000 7fe3c1e00640 1 --2- 172.21.15.28:0/1728140640 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3bc1509b0 0x7fe3bc13e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.440+0000 7fe3c1400640 1 -- 172.21.15.28:0/1728140640 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3bc1509b0 msgr2=0x7fe3bc13e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.440+0000 7fe3c1400640 1 --2- 172.21.15.28:0/1728140640 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3bc1509b0 0x7fe3bc13e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.440+0000 7fe3c1400640 1 -- 172.21.15.28:0/1728140640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3bc0c6240 msgr2=0x7fe3bc13daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.440+0000 7fe3c1400640 1 --2- 172.21.15.28:0/1728140640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3bc0c6240 0x7fe3bc13daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.440+0000 7fe3c1400640 1 -- 172.21.15.28:0/1728140640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b4009a70 con 0x7fe3bc0c1b40 2024-09-13T16:49:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.441+0000 7fe3c1400640 1 --2- 172.21.15.28:0/1728140640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0c1b40 0x7fe3bc143110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000c460 tx=0x7fe3b000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.441+0000 7fe3b9600640 1 -- 172.21.15.28:0/1728140640 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0005480 con 0x7fe3bc0c1b40 2024-09-13T16:49:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.441+0000 7fe3b9600640 1 -- 172.21.15.28:0/1728140640 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe3b00035f0 con 0x7fe3bc0c1b40 2024-09-13T16:49:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.441+0000 7fe3c2800640 1 -- 172.21.15.28:0/1728140640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3bc14d4c0 con 0x7fe3bc0c1b40 2024-09-13T16:49:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.441+0000 7fe3b9600640 1 -- 172.21.15.28:0/1728140640 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b00039c0 con 0x7fe3bc0c1b40 2024-09-13T16:49:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.441+0000 7fe3c2800640 1 -- 172.21.15.28:0/1728140640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3bc14da80 con 0x7fe3bc0c1b40 2024-09-13T16:49:14.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.442+0000 7fe3c2800640 1 -- 172.21.15.28:0/1728140640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3bc0c2a00 con 0x7fe3bc0c1b40 2024-09-13T16:49:14.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.443+0000 7fe3b9600640 1 -- 172.21.15.28:0/1728140640 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe3b0003b20 con 0x7fe3bc0c1b40 2024-09-13T16:49:14.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.444+0000 7fe3b9600640 1 --2- 172.21.15.28:0/1728140640 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe39c06ac00 0x7fe39c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:14.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.444+0000 7fe3b9600640 1 -- 172.21.15.28:0/1728140640 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fe3b0089190 con 0x7fe3bc0c1b40 2024-09-13T16:49:14.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.445+0000 7fe3c0a00640 1 --2- 172.21.15.28:0/1728140640 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe39c06ac00 0x7fe39c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:14.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.445+0000 7fe3c0a00640 1 --2- 172.21.15.28:0/1728140640 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe39c06ac00 0x7fe39c06d0c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc14cfb0 tx=0x7fe3a8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:14.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.449+0000 7fe3b9600640 1 -- 172.21.15.28:0/1728140640 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe3b0054040 con 0x7fe3bc0c1b40 2024-09-13T16:49:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:14.633+0000 7fe3c2800640 1 -- 172.21.15.28:0/1728140640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe3bc06ae30 con 0x7fe3bc0c1b40 2024-09-13T16:49:15.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.638+0000 7fe3b9600640 1 -- 172.21.15.28:0/1728140640 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe3b0055f70 con 0x7fe3bc0c1b40 2024-09-13T16:49:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-13T16:49:15.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.640+0000 7fe3c2800640 1 -- 172.21.15.28:0/1728140640 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe39c06ac00 msgr2=0x7fe39c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:15.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.640+0000 7fe3c2800640 1 --2- 172.21.15.28:0/1728140640 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe39c06ac00 0x7fe39c06d0c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc14cfb0 tx=0x7fe3a8009040 comp rx=0 tx=0).stop 2024-09-13T16:49:15.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.640+0000 7fe3c2800640 1 -- 172.21.15.28:0/1728140640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0c1b40 msgr2=0x7fe3bc143110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:15.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.640+0000 7fe3c2800640 1 --2- 172.21.15.28:0/1728140640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0c1b40 0x7fe3bc143110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000c460 tx=0x7fe3b000c920 comp rx=0 tx=0).stop 2024-09-13T16:49:15.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.641+0000 7fe3c2800640 1 -- 172.21.15.28:0/1728140640 shutdown_connections 2024-09-13T16:49:15.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.641+0000 7fe3c2800640 1 --2- 172.21.15.28:0/1728140640 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7fe39c06ac00 0x7fe39c06d0c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:15.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.641+0000 7fe3c2800640 1 --2- 172.21.15.28:0/1728140640 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3bc1509b0 0x7fe3bc13e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:15.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.641+0000 7fe3c2800640 1 --2- 172.21.15.28:0/1728140640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3bc0c6240 0x7fe3bc13daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:15.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.641+0000 7fe3c2800640 1 --2- 172.21.15.28:0/1728140640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0c1b40 0x7fe3bc143110 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:15.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.641+0000 7fe3c2800640 1 -- 172.21.15.28:0/1728140640 >> 172.21.15.28:0/1728140640 conn(0x7fe3bc0ca0e0 msgr2=0x7fe3bc156990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:15.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.641+0000 7fe3c2800640 1 -- 172.21.15.28:0/1728140640 shutdown_connections 2024-09-13T16:49:15.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.641+0000 7fe3c2800640 1 -- 172.21.15.28:0/1728140640 wait complete. 2024-09-13T16:49:15.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T16:49:15.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.824+0000 7efe2d800640 1 Processor -- start 2024-09-13T16:49:15.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.824+0000 7efe2d800640 1 -- start start 2024-09-13T16:49:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.826+0000 7efe2d800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28158b20 0x7efe280c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.826+0000 7efe2d800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe280c6a00 0x7efe280c6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.826+0000 7efe2d800640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2815c340 con 0x7efe28152f20 2024-09-13T16:49:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.826+0000 7efe2d800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe28159b50 con 0x7efe280c6a00 2024-09-13T16:49:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.826+0000 7efe2d800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe28158fd0 con 0x7efe28158b20 2024-09-13T16:49:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.826+0000 7efe27400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe280c6a00 0x7efe280c6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.826+0000 7efe1fe00640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28158b20 0x7efe280c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.826+0000 7efe27400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe280c6a00 0x7efe280c6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35752/0 (socket says 172.21.15.28:35752) 2024-09-13T16:49:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.826+0000 7efe27400640 1 -- 172.21.15.28:0/1888904261 learned_addr learned my addr 172.21.15.28:0/1888904261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:49:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.826+0000 7efe26a00640 1 --1- 172.21.15.28:0/1888904261 >> v1:172.21.15.191:6789/0 conn(0x7efe28152f20 0x7efe2815c7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.28:41042/0 (socket says 172.21.15.28:41042) 2024-09-13T16:49:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.826+0000 7efe27400640 1 -- 172.21.15.28:0/1888904261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28158b20 msgr2=0x7efe280c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:49:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.826+0000 7efe27400640 1 --2- 172.21.15.28:0/1888904261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28158b20 0x7efe280c5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.826+0000 7efe27400640 1 -- 172.21.15.28:0/1888904261 >> v1:172.21.15.191:6789/0 conn(0x7efe28152f20 legacy=0x7efe2815c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.826+0000 7efe27400640 1 -- 172.21.15.28:0/1888904261 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe281592b0 con 0x7efe280c6a00 2024-09-13T16:49:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.827+0000 7efe27400640 1 --2- 172.21.15.28:0/1888904261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe280c6a00 0x7efe280c6400 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efe20009e50 tx=0x7efe2002a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e7c66abb8a2d21 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.828+0000 7efe26000640 1 -- 172.21.15.28:0/1888904261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe20031600 con 0x7efe280c6a00 2024-09-13T16:49:15.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.828+0000 7efe26000640 1 -- 172.21.15.28:0/1888904261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7efe2002d050 con 0x7efe280c6a00 2024-09-13T16:49:15.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.828+0000 7efe26000640 1 -- 172.21.15.28:0/1888904261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe200319c0 con 0x7efe280c6a00 2024-09-13T16:49:15.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.828+0000 7efe2d800640 1 -- 172.21.15.28:0/1888904261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe280c6a00 msgr2=0x7efe280c6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.828+0000 7efe2d800640 1 --2- 172.21.15.28:0/1888904261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe280c6a00 0x7efe280c6400 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efe20009e50 tx=0x7efe2002a7c0 comp rx=0 tx=0).stop 2024-09-13T16:49:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.828+0000 7efe2d800640 1 -- 172.21.15.28:0/1888904261 shutdown_connections 2024-09-13T16:49:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.828+0000 7efe2d800640 1 --2- 172.21.15.28:0/1888904261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe280c6a00 0x7efe280c6400 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.828+0000 7efe2d800640 1 --2- 172.21.15.28:0/1888904261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28158b20 0x7efe280c5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.828+0000 7efe2d800640 1 -- 172.21.15.28:0/1888904261 >> 172.21.15.28:0/1888904261 conn(0x7efe28066040 msgr2=0x7efe28069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.828+0000 7efe2d800640 1 -- 172.21.15.28:0/1888904261 shutdown_connections 2024-09-13T16:49:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.828+0000 7efe2d800640 1 -- 172.21.15.28:0/1888904261 wait complete. 2024-09-13T16:49:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.830+0000 7efe2d800640 1 Processor -- start 2024-09-13T16:49:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.830+0000 7efe2d800640 1 -- start start 2024-09-13T16:49:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.831+0000 7efe2d800640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe280c6a00 0x7efe2814ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.831+0000 7efe2d800640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28152f20 0x7efe2815afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.831+0000 7efe2d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe28158b20 0x7efe2814d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:15.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.831+0000 7efe2d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2814f270 con 0x7efe28158b20 2024-09-13T16:49:15.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.831+0000 7efe2d800640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe281ef470 con 0x7efe280c6a00 2024-09-13T16:49:15.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.831+0000 7efe2d800640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe281ef750 con 0x7efe28152f20 2024-09-13T16:49:15.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.831+0000 7efe27400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe28158b20 0x7efe2814d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:15.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.831+0000 7efe27400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe28158b20 0x7efe2814d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35758/0 (socket says 172.21.15.28:35758) 2024-09-13T16:49:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.831+0000 7efe27400640 1 -- 172.21.15.28:0/4236707818 learned_addr learned my addr 172.21.15.28:0/4236707818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T16:49:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.831+0000 7efe26a00640 1 --2- 172.21.15.28:0/4236707818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe280c6a00 0x7efe2814ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.831+0000 7efe1fe00640 1 --2- 172.21.15.28:0/4236707818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28152f20 0x7efe2815afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.832+0000 7efe27400640 1 -- 172.21.15.28:0/4236707818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28152f20 msgr2=0x7efe2815afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.832+0000 7efe27400640 1 --2- 172.21.15.28:0/4236707818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28152f20 0x7efe2815afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.832+0000 7efe27400640 1 -- 172.21.15.28:0/4236707818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe280c6a00 msgr2=0x7efe2814ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.832+0000 7efe27400640 1 --2- 172.21.15.28:0/4236707818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe280c6a00 0x7efe2814ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.832+0000 7efe27400640 1 -- 172.21.15.28:0/4236707818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe20009b30 con 0x7efe28158b20 2024-09-13T16:49:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.832+0000 7efe1fe00640 1 --2- 172.21.15.28:0/4236707818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28152f20 0x7efe2815afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:49:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.832+0000 7efe26a00640 1 --2- 172.21.15.28:0/4236707818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe280c6a00 0x7efe2814ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:49:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.832+0000 7efe27400640 1 --2- 172.21.15.28:0/4236707818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe28158b20 0x7efe2814d5e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efe20009e50 tx=0x7efe20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.832+0000 7efe1ea00640 1 -- 172.21.15.28:0/4236707818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2003b600 con 0x7efe28158b20 2024-09-13T16:49:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.832+0000 7efe1ea00640 1 -- 172.21.15.28:0/4236707818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7efe2003b760 con 0x7efe28158b20 2024-09-13T16:49:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.832+0000 7efe1ea00640 1 -- 172.21.15.28:0/4236707818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2003bb10 con 0x7efe28158b20 2024-09-13T16:49:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.832+0000 7efe2d800640 1 -- 172.21.15.28:0/4236707818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe281efa30 con 0x7efe28158b20 2024-09-13T16:49:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.833+0000 7efe2d800640 1 -- 172.21.15.28:0/4236707818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe281efff0 con 0x7efe28158b20 2024-09-13T16:49:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.833+0000 7efe2d800640 1 -- 172.21.15.28:0/4236707818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2814d300 con 0x7efe28158b20 2024-09-13T16:49:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.834+0000 7efe1ea00640 1 -- 172.21.15.28:0/4236707818 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efe2003bc70 con 0x7efe28158b20 2024-09-13T16:49:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.835+0000 7efe1ea00640 1 --2- 172.21.15.28:0/4236707818 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7efe0406ac00 0x7efe0406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:49:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.835+0000 7efe1ea00640 1 -- 172.21.15.28:0/4236707818 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5817+0+0 (secure 0 0 0) 0x7efe200aa190 con 0x7efe28158b20 2024-09-13T16:49:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.836+0000 7efe26a00640 1 --2- 172.21.15.28:0/4236707818 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7efe0406ac00 0x7efe0406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:49:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.836+0000 7efe26a00640 1 --2- 172.21.15.28:0/4236707818 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7efe0406ac00 0x7efe0406d0c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efe18007fd0 tx=0x7efe18007d40 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:49:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:15.840+0000 7efe1ea00640 1 -- 172.21.15.28:0/4236707818 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7efe2814d300 con 0x7efe28158b20 2024-09-13T16:49:16.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:16.016+0000 7efe2d800640 1 -- 172.21.15.28:0/4236707818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efe2814d300 con 0x7efe28158b20 2024-09-13T16:49:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:16.643+0000 7efe1ea00640 1 -- 172.21.15.28:0/4236707818 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7efe2814d300 con 0x7efe28158b20 2024-09-13T16:49:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-13T16:49:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:16.646+0000 7efe2d800640 1 -- 172.21.15.28:0/4236707818 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7efe0406ac00 msgr2=0x7efe0406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:16.646+0000 7efe2d800640 1 --2- 172.21.15.28:0/4236707818 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7efe0406ac00 0x7efe0406d0c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efe18007fd0 tx=0x7efe18007d40 comp rx=0 tx=0).stop 2024-09-13T16:49:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:16.646+0000 7efe2d800640 1 -- 172.21.15.28:0/4236707818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe28158b20 msgr2=0x7efe2814d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:49:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:16.646+0000 7efe2d800640 1 --2- 172.21.15.28:0/4236707818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe28158b20 0x7efe2814d5e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efe20009e50 tx=0x7efe20004a40 comp rx=0 tx=0).stop 2024-09-13T16:49:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:16.646+0000 7efe2d800640 1 -- 172.21.15.28:0/4236707818 shutdown_connections 2024-09-13T16:49:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:16.646+0000 7efe2d800640 1 --2- 172.21.15.28:0/4236707818 >> [v2:172.21.15.191:6832/1568228941,v1:172.21.15.191:6833/1568228941] conn(0x7efe0406ac00 0x7efe0406d0c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:16.646+0000 7efe2d800640 1 --2- 172.21.15.28:0/4236707818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe28158b20 0x7efe2814d5e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:16.646+0000 7efe2d800640 1 --2- 172.21.15.28:0/4236707818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28152f20 0x7efe2815afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:16.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:16.646+0000 7efe2d800640 1 --2- 172.21.15.28:0/4236707818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe280c6a00 0x7efe2814ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:49:16.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:16.646+0000 7efe2d800640 1 -- 172.21.15.28:0/4236707818 >> 172.21.15.28:0/4236707818 conn(0x7efe28066040 msgr2=0x7efe28156760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:49:16.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:16.646+0000 7efe2d800640 1 -- 172.21.15.28:0/4236707818 shutdown_connections 2024-09-13T16:49:16.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T16:49:16.646+0000 7efe2d800640 1 -- 172.21.15.28:0/4236707818 wait complete. 2024-09-13T16:49:16.671 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-13T16:49:16.671 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T16:49:16.672 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:16.687 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T16:49:16.687 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T16:49:16.687 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:16.744 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T16:49:16.744 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-09-13T16:49:16.744 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:16.799 INFO:tasks.ceph.mds.e:Stopped 2024-09-13T16:49:16.800 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T16:49:16.800 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:16.811 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T16:49:16.811 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T16:49:16.811 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:16.863 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T16:49:16.863 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-09-13T16:49:16.864 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:16.915 INFO:tasks.ceph.mds.f:Stopped 2024-09-13T16:49:16.915 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-13T16:49:16.915 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-13T16:49:16.915 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:16.964 INFO:tasks.ceph.osd.0:Stopped 2024-09-13T16:49:16.964 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-13T16:49:16.964 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:17.020 INFO:tasks.ceph.osd.1:Stopped 2024-09-13T16:49:17.020 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-13T16:49:17.020 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:17.071 INFO:tasks.ceph.osd.2:Stopped 2024-09-13T16:49:17.071 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-13T16:49:17.071 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:17.121 INFO:tasks.ceph.osd.3:Stopped 2024-09-13T16:49:17.122 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-13T16:49:17.122 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:17.153 INFO:tasks.ceph.osd.4:Stopped 2024-09-13T16:49:17.154 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-13T16:49:17.154 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:17.224 INFO:tasks.ceph.osd.5:Stopped 2024-09-13T16:49:17.224 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-13T16:49:17.225 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:17.302 INFO:tasks.ceph.osd.6:Stopped 2024-09-13T16:49:17.303 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-13T16:49:17.303 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:17.385 INFO:tasks.ceph.osd.7:Stopped 2024-09-13T16:49:17.385 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-13T16:49:17.385 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-13T16:49:17.385 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:17.449 INFO:tasks.ceph.mgr.y:Stopped 2024-09-13T16:49:17.450 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-13T16:49:17.450 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:17.482 INFO:tasks.ceph.mgr.x:Stopped 2024-09-13T16:49:17.482 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-13T16:49:17.482 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-13T16:49:17.482 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:17.501 INFO:tasks.ceph.mon.a:Stopped 2024-09-13T16:49:17.501 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-13T16:49:17.501 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:17.535 INFO:tasks.ceph.mon.b:Stopped 2024-09-13T16:49:17.535 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-13T16:49:17.535 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:49:17.588 INFO:tasks.ceph.mon.c:Stopped 2024-09-13T16:49:17.588 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-13T16:49:17.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | head -n 1 2024-09-13T16:49:17.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-09-13T16:49:17.685 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T16:49:17.686 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-13T16:49:17.962 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T16:49:17.962 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-13T16:49:18.139 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T16:49:18.140 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-13T16:49:18.241 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T16:49:18.242 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-13T16:49:18.358 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-13T16:49:18.358 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-13T16:49:18.562 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-13T16:49:18.563 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-13T16:49:18.657 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-13T16:49:18.658 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-13T16:49:18.711 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-13T16:49:18.711 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-13T16:49:18.774 INFO:tasks.ceph:Archiving mon data... 2024-09-13T16:49:18.775 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903833/data/mon.a.tgz 2024-09-13T16:49:18.775 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-09-13T16:49:18.812 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.hDMg8Zv8JQ 2024-09-13T16:49:18.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.hDMg8Zv8JQ 2024-09-13T16:49:18.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.hDMg8Zv8JQ 2024-09-13T16:49:19.083 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.hDMg8Zv8JQ is 365KB 2024-09-13T16:49:19.174 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.hDMg8Zv8JQ 2024-09-13T16:49:19.209 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903833/data/mon.b.tgz 2024-09-13T16:49:19.209 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-09-13T16:49:19.228 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.m4Sdm9SMAh 2024-09-13T16:49:19.229 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.m4Sdm9SMAh 2024-09-13T16:49:19.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.m4Sdm9SMAh 2024-09-13T16:49:19.478 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.m4Sdm9SMAh is 365KB 2024-09-13T16:49:19.574 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.m4Sdm9SMAh 2024-09-13T16:49:19.590 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903833/data/mon.c.tgz 2024-09-13T16:49:19.590 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-09-13T16:49:19.644 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.kPfZuQsMXd 2024-09-13T16:49:19.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.kPfZuQsMXd 2024-09-13T16:49:19.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.kPfZuQsMXd 2024-09-13T16:49:19.858 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.kPfZuQsMXd is 367KB 2024-09-13T16:49:19.954 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.kPfZuQsMXd 2024-09-13T16:49:19.968 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-13T16:49:19.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T16:49:19.973 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T16:49:20.033 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-13T16:49:20.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-13T16:49:20.037 DEBUG:teuthology.orchestra.run.smithi191:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-13T16:49:20.095 INFO:teuthology.orchestra.run.smithi028.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-13T16:49:20.100 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-13T16:49:20.105 INFO:tasks.ceph:Archiving crash dumps... 2024-09-13T16:49:20.107 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903833/remote/smithi028/crash 2024-09-13T16:49:20.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T16:49:20.163 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903833/remote/smithi191/crash 2024-09-13T16:49:20.164 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T16:49:20.188 INFO:tasks.ceph:Shutting down logrotate 2024-09-13T16:49:20.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:49:20.193 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:49:20.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-13T16:49:20.359 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-13T16:49:20.384 INFO:tasks.ceph:Compressing logs... 2024-09-13T16:49:20.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T16:49:20.388 DEBUG:teuthology.orchestra.run.smithi191:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T16:49:22.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-09-13T16:49:40.707 INFO:tasks.ceph:Archiving logs... 2024-09-13T16:49:40.707 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903833/remote/smithi028/log 2024-09-13T16:49:40.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T16:49:43.050 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903833/remote/smithi191/log 2024-09-13T16:49:43.051 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T16:49:45.393 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T16:49:45.403 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T16:49:45.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T16:49:45.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T16:49:45.529 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T16:49:45.538 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T16:49:45.538 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T16:49:45.543 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T16:49:45.561 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-09-13T16:49:45.565 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T16:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-09-13T16:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv01.front.sepia.ceph.com 3 6 377 44 +189us[ +192us] +/- 38ms 2024-09-13T16:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 43 +257us[ +257us] +/- 69ms 2024-09-13T16:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 -621us[ -618us] +/- 71ms 2024-09-13T16:49:45.566 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T16:49:45.574 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-09-13T16:49:45.581 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T16:49:45.581 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-09-13T16:49:45.581 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv01.front.sepia.ceph.com 3 6 377 42 +188us[ +186us] +/- 38ms 2024-09-13T16:49:45.581 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 44 +249us[ +248us] +/- 69ms 2024-09-13T16:49:45.582 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 -625us[ -625us] +/- 71ms 2024-09-13T16:49:45.582 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T16:49:45.583 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T16:49:45.592 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T16:49:45.593 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T16:49:45.602 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T16:49:45.745 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T16:49:45.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T16:49:45.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:49:45.875 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T16:49:45.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:49:45.905 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T16:49:45.915 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T16:49:45.926 INFO:teuthology.task.internal:Duration was 789.692848 seconds 2024-09-13T16:49:45.927 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T16:49:45.935 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T16:49:45.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T16:49:45.940 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T16:49:45.986 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T16:49:46.023 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T16:49:46.486 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T16:49:46.487 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T16:49:46.487 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T16:49:46.536 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2024-09-13T16:49:46.536 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T16:49:46.564 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T16:49:46.564 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T16:49:46.568 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T16:49:46.633 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T16:49:46.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T16:49:46.637 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T16:49:46.780 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T16:49:46.790 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T16:49:46.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T16:49:46.847 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T16:49:46.872 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T16:49:46.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:49:46.892 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:49:46.935 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2024-09-13T16:49:46.958 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-09-13T16:49:46.988 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:49:47.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:49:47.022 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:49:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:49:47.036 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T16:49:47.049 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T16:49:47.050 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903833/remote/smithi028 2024-09-13T16:49:47.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T16:49:47.126 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903833/remote/smithi191 2024-09-13T16:49:47.127 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T16:49:47.159 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T16:49:47.160 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T16:49:47.164 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T16:49:47.211 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T16:49:47.223 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T16:49:47.223 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T16:49:47.244 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T16:49:47.244 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T16:49:47.248 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T16:49:47.266 INFO:teuthology.orchestra.run.smithi191.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 16:49 /home/ubuntu/cephtest 2024-09-13T16:49:47.279 INFO:teuthology.orchestra.run.smithi028.stdout: 393255 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 16:49 /home/ubuntu/cephtest 2024-09-13T16:49:47.281 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T16:49:47.292 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T16:49:47.315 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 2ca44a2f35ca8f36983aaaa46d49312239ef519a description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} duration: 789.6928477287292 flavor: default owner: scheduled_rishabh@teuthology success: true 2024-09-13T16:49:47.315 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T16:49:47.409 INFO:teuthology.run:pass